King – Man + Woman ≈ King

Posted by on November 19, 2024 No comments

An article by Plotly1 shows how the main analogy for what embeddings represent is a bit shaky. It sounds good and works like the analogies logic from the SAT. But it only kind of works.

I was reading all this stuff about embeddings and it’s really a surprising feature that you could do simple vector math to solve analogies. At first I assumed that the major trained models must have been trained with analogies. You really could create layers that subtract and add embeddings and do supervised learning on a known set of analogies, training the embeddings to represent gender with a single, similar vector. But nobody does this. It seems like just an amazing emergent feature of training the hidden layer we get the embeddings from!

It kind of works. But not quite, and it takes a bit of cheating.2 Taking an embedding vector for King, subtracting the vector for Man and adding the vector for Woman makes a new vector. But the cosine distance isn’t far enough away from King. You have to exclude King, Man, and Woman in order to make this vector close to the embedding vector for Queen.

I tried this myself with OpenAI’s modern embedding model, “text-embedding-3-small”.3 It gives the same results. The closest vector is still King. There’s a lot of error even if you exclude King from the possible results. The embeddings themselves don’t store good analogy information in just linear vector math. The hidden layers and attention heads will still have to do to make the embeddings into something really useful. It’s neat that we get somewhat close values and it shows something is encoded in these embeddings. The results just don’t work exactly as I’ve been reading and what I’ve been taught about embeddings.

This slightly disappointing result intuitively make sense to me. The model wasn’t trained to make vector addition a feature of its values. It’s a bunch of information ready for transfer learning. And what other information is in there? Embedding vectors encode a lot more information, maybe important nuances and maybe extraneous values, in the vector values than you would want in a space made for analogies.

  1. Plotly Graph, “Understanding Word Embedding Arithmetic: Why there’s no single answer to “King − Man + Woman = ?”. https://medium.com/plotly/understanding-word-embedding-arithmetic-why-theres-no-single-answer-to-king-man-woman-cd2760e2cb7f, 2020 ↩︎
  2. Florian Huber, “King – Man + Woman = King ?”. https://blog.esciencecenter.nl/king-man-woman-king-9a7fd2935a85, 2019 ↩︎
  3. Code to try this with OpenAI’s embeddings: https://github.com/seanmcnealy/seanmcnealy_samples/blob/master/embeddings.py ↩︎

Software Books

Posted by on November 8, 2024 No comments

I know there’s a lot of reviews out there. And hopefully you already have these books if you work or intend to work in the software industry.

Designing Data-Intensive Applications by Martin Kleppmann

A comprehensive book on actually using distributed systems. You’ll want to know what the effects are from your design choices. And this book goes into databases based on the experiences of someone that’s done this work before you. I read this after prototyping a system, and I think the content here is worth about six months of full time work and research. Plus he has citations if you want to go more in depth.

Refactoring by Martin Fowler

Software can change over time and you can change your mind about design choices as the solution changes. Functional or Object-Oriented or IOC or visitor pattern, this walks through practical mechanical steps to show how easy it is to make changes that simplify programs for readability and maintenance. And shows how to go back the other way, too, because maybe that way is better.

You’ll see your programs with many more opportunities to express your designs, and it’ll help you choose for better code.

And I’ll conclude with a short list of books you’ll have to one day see if you study software, but are much less practical. The Art of Computer Programming volumes 1 – 4b, the “CLRS” algorithms book, the compilers “Dragon book.” You’ll have to have them, I guess.

Software Engineering Licensure

Posted by on April 12, 2024 No comments

In the US, there is a software engineering license. Is it worth it to become licensed? Is it it feasible?1 No! It’s neither worth it nor feasible.

Engineered products and designs need to show they meet standards and will not harm people. Engineering licensure is one of the activities in some fields that shows expertise at least of a required level to make a correct design. It’s widely used in building design. Licensure is much less used in electrical, computer, or software design. Can a software engineer get licensed this way? A Professional Engineering license can be given in the specialization of Electrical and Computer Engineering.2 That’s pretty close to software development. It may seem like this could be used to make software products more dependable and reliable.

There are general requirements to become licensed. In the US, licensing is done by each state, and every state follows the basic model of engineering licensure (called the Model Law by NCEES), with education, experience, and examination. This is straightforward for some engineers, but difficult for computing professionals. Here are the three major steps and why each is difficult.

Education – A four year degree from an ABET accredited university. IT and CS degrees are beginning to become accredited by ABET, though through their Computing Accreditation Commission. The education piece requires ABET Engineering Accreditation Commission, so many degrees do not count for the education component. Many of the best programs in our field, like Stanford or MIT, do not have this.

Experience – Four years of experience working under a licensed professional engineer. This is rare in industries either under the industrial exemption or under federal certification. Industrial exemption is a rule where a corporation takes responsibility and liability for designs, errors, and omissions and the designer does not require a professional license. Defense, aerospace, medical, even electrical other than power have few licensed practicing engineers. To show how few: there were 13 people passing the Electrical and Computer exam in 2022.3 If you work in computing software or hardware, odds are you do not work directly for one of those 13 people.

Examination – The FE exam and PE exam. The FE exam is so broad and difficult. Computing has moved so fast that even the foundations targeted in the PE exam are difficult to keep constant and stay relevant. Eight hour exams are always difficult.

The results could be considered underwhelming at best. The exam doesn’t prove you’re good at computing work. The requirements for education and experience aren’t a discriminating factor between other education and other experience that turn out to be just as good as what’s accepted for engineering. There’s no guaranteed or even defined career advancement or PE only work. A license is just symbolic of professionalism.

And so, against all the difficulties, and the almost no payoff, I did it. I got a PE license. And this is how.

I went to college at the University of Florida where I had chosen to graduate through the College of Engineering. The Computer Engineering degree program at UF is ABET EAC accredited. No one is choosing computing programs based on this, and I would have happily gone to a school that did not have this feature. Choosing Computer Science or Information Technologies majors would have been a different college than engineering, and would have also worked well for me in every aspect except for this license.

Examination was very difficult. I studied a lot for the FE exam. When I took it in 2008 there was a breadth component that covered all of engineering. Topics range from structural forces to materials to circuits. The PE exam I found easier, but was still only offered once a year in 2023 due to fewer people taking the computing exam.

Experience is basically impossible. But there are exceptions depending on the state board. Alabama will consider whether your supervisor hypothetically could also apply. Washington will consider work for a federal entity or manufacturing as suitable experience (think of Boeing and Lockheed). Florida will use a diploma or copy of official transcripts from your supervisor only if your supervisor has a degree in engineering (think of NASA, Northrop, and L3 Harris). I had no trouble finding PEs in my network of people I know that were willing to recommend me. But I had to rely on Florida’s industrial exemption for my supervisors. If I had lost contact or any of these supervisors were unwilling to help so much or had passed, this would not have been possible.

15 years after I took the FE exam and 4 months after sending in my application, I almost gave up when I had to request an official transcript sent for a retired supervisor who had graduated 50 years prior in another country. I was surprised the university would even do it. University of Alberta was happy to help, but they couldn’t use the newer, faster system. We waited another 3 weeks for that paperwork in the mail.

And there you have it. I can’t recommend anyone else do this. Licensure requires a lot of luck and a lot of following the rules and a lot of waiting. Each of education, experience, and examination excludes too many talented and hard working people for the software field to consider it seriously. On the positive, there are some amazing people working on this behind the scenes. People at state boards, NCEES, IEEE, ABET, and professors at engineering colleges are working hard and doing smart things. They are bringing Professional Engineering forward and making it better for all involved, and including the electronics and computing professionals along the way when they can. Software, hardware, and computing could use some more licensing, or credentialing, or product/process certification than it has today to catch up with other engineering disciplines that are entrusted with safety critical designs and verification, but licensing isn’t ready for that yet.

  1. Glithero, Jason. “Can we strengthen the Professional Engineer (PE) License.” https://www.linkedin.com/pulse/can-we-strengthen-professional-engineer-pe-license-glithero-p-e-/ ↩︎
  2. Musselman, Craig, et al. “A Primer on Engineering Licensure in the United States.” https://www.nspe.org/sites/default/files/resources/pdfs/blog/ASEE-A-Primer-on-Engineering-Licensure-in-the-United-States.pdf ↩︎
  3. NCEES. “Squared.” 2023. https://ncees.org/wp-content/uploads/2024/03/Squared-2023_web-1.pdf ↩︎

Digital Nomads Part 4

Posted by on February 18, 2024 No comments

Costs

I spent 5 weeks in Spain. There’s some benefits to choosing Spain, as it has a quite favorable cost for rent and food, while still being totally modern. See the other posts for thoughts and advice. This one is just numbers.

Rent$3,300
Hotels$1,700
Airfare$3,800
Airfare for Tourism$1,400
Food and Restaurants$4,000
Costs at Home$500

There is more money spent on tourism things. Turns out if you aren’t in Europe often and you’re very close to Paris and London that it’s very tempting to go visit. This added a lot of costs that could have been limited. But watch out, if you have the ability you’ll probably spend a lot in this category.

Digital Nomads Part 3

Posted by on December 31, 2023 No comments

Time Zones

The most obvious issue. It’s challenging to work several hours from where your coworkers work. Many of us have worked with remote teams, for example Hyderbad or Prague. There’s people that take meetings very early or very late their local time, and they do it every day. That doesn’t make it easy.

The city you’re living in may have more defined hours than in the United States. The rest of the world doesn’t quite have the 24 hour culture of availability. It’s easy to get a Chipotle burrito at noon, 2pm, 4pm, or 8pm. But there may be smaller ranges of hours these things are available. So if you want lunch, let your coworkers know what hours you are taking off for lunch.

Work Expectations

Set some goals ahead of time. Ask your boss what you should be working on.

When you’re far away and some time zones different is a great time to accomplish some of those tasks that you can do alone. Like some of those things you’ve been putting off when new problems get brought to your desk at work to deal with immediately. Think up some tech debt or a cool new feature that you’ve been meaning to finish when you get some time.

Just make sure it’s aligned with what your company, managers, and team members want from you.

Travel Planning

Do a lot of this ahead of time. You will be able to take some improvised trips to nearby attractions. But if things aren’t already planned you’re taking away time from your temporary home to plan things. So have the big things ready to go.

Do you want to take a train or plane to another nearby city? Have the transportation and hotels planned. Maybe even a day trip or museum tickets. Leave some room for spontaneous decisions, but having to work and be a travel agent takes up a lot of time.

Electrical Connectors

Your technology runs on different connectors than the rest of the world. Most AC/DC adapters, like a phone charger or laptop power supply, work on almost every power supply. You just need a safe way plug in a connection. Bring an adaptor everywhere and look up which ones you’ll need in different countries.

What it’s all like

It’s amazing. It’s tiring sometimes. You’ll get a bunch of stories. It’s worth it.

Meeting People

You’re not on study abroad anymore. This is tough now. It’s only getting tougher. We all have technology that connects us to the people we already know. If you’re traveling full time you have the energy and time and ability to seek out those places and times you can meet locals or other travelers. If you’re working and spend many hours in meetings or programming, you’re connected to home instead of where you’ve traveled.

I don’t have a great answer to this. But make sure when you put the laptop away that you’re off of work time. It’s worth it to make that clear boundary even if you don’t define work and home time as much at home.

Digital Nomads Part 2

Posted by on October 2, 2023 No comments

Putting your home on hold

You live somewhere. And you’ll be pretty far away from that place for a long time.

It may be surprisingly difficult to use some internet services. Logins that are simple from home may ask for additional information when accessing your account from another continent. This gets even more difficult if you’re using a different local phone number. Most of these you won’t know about until you try to use them.

See if you can access your texts or voicemails from somewhere else than your phone.

Get rent or mortgate, utilities, insurance, and any other bills onto autopay before going anywhere. I have my bank lookup utility bills and send payment rather than the utility just taking money out of my account. In the end, that just works the same way.

Pets

You’ll miss your furry friends and they’ll miss you. Arrange for someone to send you pictures to remind you they’re doing alright. And leave a way to get them some treats.

Neighbors

Have some people nearby that you can trust with your home. Whether you’re in a condo building or live on acres, looking out for each other is what makes everything work.

Someone coming by once in a while to check on things can let you know about leaks, flooding, failing equipment, and any other issues.

Home Automation

A lot of peace of mind can come from a couple of cameras. Reolink makes a few that aren’t subscription services. But the Google and Amazon devices work great too.

And finally, you may not be able to put everything on hold. Life happens. Hopefully you don’t have an issue like a family emergency call you back home. But have enough money in a savings account to get yourself home on relatively short notice.

Digital Nomads Part 1

Posted by on September 19, 2023 No comments

You’ve noticed that as a programmer you can work from anywhere. So how do you begin checking in to your daily standup from Starbucks in Paris?

Passport

If you don’t have a passport, get one today. Even if you’re not going anywhere soon. US passports are good for about 10 years. They can take three to six months to get, though. So start today before you need one. It just takes a few documents copied and get your photos taken.

Passport photos have special requirements. But there’s no need to pay $15 – $20 at the drug store. Turns out they just use your phone to take a photo now, which you can do yourself. Walmart will print a 4″x6″ matte photo for $0.12 and you can pick it up almost immediately. I recommend IDPhoto4You to adjust the image. You may need a few of these photos. Keep some around for various government services. You may even use them for city transit cards.

Visas

Easiest way to get into a country is visa free as a tourist. If you have a return flight already booked and know where you’ll be staying then you can usually stay 90 days without any trouble or extra paperwork at all. The different ways to do this can have a huge depth. Passport Index is a great way to start seeing where you can go without a visa, where you can apply when you arrive, and where you’ll need to apply ahead of time.

Always check on any additional requirements, though the airlines are pretty good at pointing you in the right directions, the responsibility lies on you to know what’s going on. There were additional forms for COVID in 2022, but those were generally gone by 2023.1

Digital Nomad Visas

A lot of countries are making it easier for remote workers to stay even longer. Long stay visas, digital nomad visas, and temporary worker visas all run a spectrum of different ways belong in a country.2 Many give you access to other EU countries and some can start the clock on time for attaining a new citizenship. Some allow benefits for immediate family and some allow access to the school system for children.

For short stay visas you may be able to get away with only paying income taxes in your home country. Some digital nomad visas won’t require full income tax where you’re staying, but do require something back for what they’re offering. The longer and more permanent your stay, the more you will be expected to pay taxes. And the US always taxes its citizens, even abroad.

Driving

Many countries will allow you to use your normal driver’s license. Some require and some recommend the use of a translated license.3 In the US, this is a document you can get from a AAA office by showing your driver’s license and giving them two of those extra passport photos. It is good for one year, but you need to get it before traveling.

Public Transport

Many public transport systems have their own cards. If this is available before you leave, go ahead and get one. Usually they are available only in person and will require a photograph or another one of those passport photos you are carrying. Local transport cards can save you a lot when you purchase by the month rather than the tourist rates.

For example, Madrid’s 30 day pass was €21.80 in 2023.4 Far less than paying by the trip. But they only mail the card within the city, so make an appointment to pick one up after you arrive. But don’t worry in London, where you can just use any credit card to tap into train stations or onto busses.

Money

In 2023 most highly developed or touristy places will just prefer you use your Visa or Mastercard. You may not need much paper currency or coins at all. Having a Revolut debit/ATM card can help you avoid bank fees.

Try to always pay in the local currency.5 Most payment terminals and ATMs may ask if you want to exchange into your home currency. Especially the ATMs you may even have to reject their conversion before moving on. This conversion is generally way worse than what your bank would offer.

  1. U.S. Ends Last Covid Travel Barrier, Vaccine Mandate for Foreign Arrivals, NYTimes, https://www.nytimes.com/2023/05/12/travel/covid-vaccine-mandate-us.html, 2023 ↩︎
  2. 66 Digital Nomad Visa Countries in 2023, CitizenRemote, https://citizenremote.com/blog/digital-nomad-visa-countries/#:~:text=Europe%20Digital%20Nomad%20Visa%20Countries, 2023 ↩︎
  3. Geneva Convention on Road Traffic 1949 https://en.wikipedia.org/wiki/Geneva_Convention_on_Road_Traffic and Vienna Convention on Road Traffic 1968 https://en.wikipedia.org/wiki/Vienna_Convention_on_Road_Traffic ↩︎
  4. Madrid Temporary Discounts, https://www.metromadrid.es/es/descuentostemporales, 2023 ↩︎
  5. How to manage your money as a digital nomad, Lonely Planet, https://www.lonelyplanet.com/articles/managing-finances-digital-nomad-remote-work#:~:text=Credit%20and%20debit%20cards, 2021 ↩︎

C for Programmers

Posted by on April 9, 2017 No comments

There are programmers who have never used C before! This is a shocking fact to me, as C is where I started programming. And there are many programmers who want to learn to use C, but already know how to program. C for Beginners is easy to find with a Google search. They need an introduction to pointers and a bit of malloc. Hopefully without mentioning things like functions, types, and how to write a conditional which would surely put a ruby, python, or java programmer to sleep.

So here it is. A few code examples with pointers. I mention tools like gcc and some flags and gdb. And there is some assembly code, as really writing C requires some computer architecture understanding.

PDF

Or here’s a document with the same goals from Stanford: Stanford CS Education Library: 102

Ad test

Posted by on May 13, 2016 No comments

I’m working on a advertising deliverable for a client. I need to show the differences between something like Google’s AdSense and statically delivering image content. Because of this, here is an Ad.

It’s a 250x250px ad, with backup of Jake the dog, http://www.otherroute.net/wordpress/wp-content/uploads/2016/05/ad_test.html.

This took me 3 days to get approval to get working. First day is submitting the site and Google crawls it to verify you have enough content. My first domain didn’t pass (but it’s basically just a business card). This one did. Then I placed the ad code using wordpress plugin “Easy Plugin for AdSense”. It was blank (with no way to get anything other than blank) for about 48 hours.

I’ll get rid of this later. I have no aspirations about making some glorious $0.05 or whatever small personal blogs can make.

update: Removed! It shows that I made $0.00. I thought it would make a couple more cents. It was easy to do though, and if you have a specialized enough userbase I’m sure Google is a good partner today for ads.

My System 76 laptop, cheap, rattling, smoking, all around pretty good

Posted by on June 15, 2015 1 comment

One year ago I bought a System 76 Gazelle laptop. It’s a pretty good machine. I’d say only for people that can tinker with their laptop though.

Here’s the order specs:

Gazelle Professional ( gazp9 ) Quantity: 1
Ubuntu 14.04 LTS 64 bit
15.6″ 1080p Full High Definition Display with Matte Surface ( 1920 x 1080 )
Intel® High Definition Graphics 4600
4th Generation Intel® Core™ i7-4810MQ Processor ( 2.8 GHz 6MB L3 Cache – 4 Cores plus Hyperthreading )
16 GB Dual Channel DDR3 SDRAM at 1600MHz – 2 X 8 GB
United States Keyboard Layout
500 GB Samsung 840 EVO Solid State Drive
No Secondary Hard Drive
Caddy Case without Hard Drive (Use your own drive in the optical bay)
Intel® Dual Band Wireless-AC 3160 AC Wireless LAN up to 433 Mbps + Bluetooth
1 Yr. Ltd. Warranty and 1 Yr. Technical Support
Rush Assembly Service – Guaranteed shipping within 2 business days of your order ( not including day of order )

All this came to a bit more that $1500, just about the same price as it would be today. That’s a hell of a deal! Go price out a Macbook with a 4810MQ, 16GB RAM, and 500 GB SSD. Or don’t because I know that price. It’s the $2500 one. The Macbook does have a better screen, incredible touchpad, and you can add in a 2GB AMD video module for practically nothing at that level. The Apple has no Ethernet or VGA port.

So for a work laptop that won’t run better with the improved video and will be plugged in to an external keyboard, mouse, and monitor this is a no brainer. I actually prefer Ubuntu because the package management is just so much easier on Linux than anywhere else. Java development is cross platform so I’m not missing out on anything there.

Anyway, so what’s the bad side? Some screws inside were loose from the factory. The keyboard felt weak and weird. Then there was some rattling around when I would move the laptop. Opened it up and a bunch of screws fell out. Tightening them down fixed both problems! Just sad they missed that step when building the machine.

Next happened just outside the 1 year warrenty. Smoke started coming out the rear left near the power port. I took it apart! The power port has 2 tabs on the back, one connected to the port’s housing and the other directly to the plug. The solder joint from there to wire to the motherboard was bad and had broken off the tab it was supposed to connect to. But it was still good on the other tab. Oh no! The ground for all the power routed around the port’s casing, finding resistance and heating up enough to melt the plastic case some. It was pretty easy to remove screws and solder that back together. Points for maintainability. If something is going to be produced poorly enough to do this it had better be fixable.

So all in all, a pretty good laptop. Parts are easy to access and change or upgrade. Ubuntu is great. Etherenet, VGA, HDMI, USB is all there. A good laptop for a developer. It does require at least knowing someone who can operate a soldering iron.