Monday, September 19, 2016

Progress on suneido.js

A project like suneido.js goes through distinct stages.

The first stage is just thinking and researching the platform. (In this case JavaScript, ES6/2015, TypeScript, node.js etc.) I've been doing that in the background for quite a while.

But eventually you have to start writing some actual code, and that's what I've been doing lately. This can be a frustrating stage because it's three steps forward, two steps back (and that’s on a good day). You pick an area to implement, decide on an approach, and write some code. That often goes smoothly. But then you start on the next area and you realize that the approach you picked for the first area won't work for the next. So you come up with an approach that works for both areas and go back and rewrite the existing code. Rinse and repeat.

Progress seems painfully slow during this stage. Important advances are being made, it’s just that it’s in terms of design decisions, not lines of code. (On the positive side, the Suneido language is, in many ways, quite similar to JavaScript, so the impedance mismatch is much lower than with C++ or Java. Meaning I can map many features quite directly from one to the other.)

There's also learning involved since I'm on a new platform, with a new language and tools. I went through the same thing with jSuneido since I'd never worked in Java before I started that project. It's one thing to read books about languages and platforms. It's a whole 'nother story using them to implement a large project. (I have to say, ES6/2015 features have been a big plus on this project, as has TypeScript.)

Eventually the approach firms up and all of a sudden you can make rapid progress in the code. It almost becomes mechanical, simply translating from jSuneido or cSuneido. Of course, issues still arise as you encounter different corners and edge cases. But mostly they are easily addressed.

It reminds me a bit of search strategies like simulated annealing where you start by making large changes all over, and as you get closer to a solution, you “cool down” and the changes are smaller and smaller as you approach a solution. Of course, it doesn’t mean you’ve found the “best” solution. But hopefully the initial larger explorative jumps covered enough of the search space that you’ve found something reasonably close to optimal.

I’m always surprised by just how many dead ends there are. Of course, when you’re programming the possibilities are infinite, so a lot of them have to be wrong. On the other hand, there is supposed to be intelligent design at work here, which you'd think would avoid so many dead ends. But many things are hard to evaluate before you actually try them. (Which brings to mind Wolfram’s idea of computational irreducibility.)

The reward for this plodding is that seemingly all of a sudden, I have a system that can actually handle real problems, not just limited toy examples. There is still lots to do, but suneido.js can now successfully run some of Suneido's standard library tests. It takes a surprising amount of infrastructure to reach this stage. Even “simple” tests tend to use a lot of language features.This seems like a big leap forward, but I know from implementing jSuneido that it’s really just another small step in a long process.

It's a bit hard to follow the project I'm afraid. The TypeScript code is on GitHub, but the actual transpiler is written in Suneido code and lives on our private version control system. And running the system requires jSuneido (which is used to parse Suneido code to an AST) and cSuneido (for the IDE), plus node.js and Typescript and other bits and pieces. I really should try to simplify or at least package it so it's easier for other people to access.

I'm glad to have reached this (positive) point, since I’m heading off travelling again, and probably won’t get too much done on this for a while.

PS. Hearing about some paragliding deaths recently I thought (not for the first time), "what if I get killed paragliding?". And my first thought was, "but who would finish suneido.js?". Ah, the mind of a true geek :-)

Friday, September 02, 2016

Old School

As I progress on suneido.js I'm accumulating a number of miscellaneous commands I need to run regularly. So far I've avoided any kind of build tool. I started writing shell scripts (OS X) and batch files (Windows) which worked but is ugly. And led to git problems with executable permissions.

So I started looking at build tools. The big one these days seems to be Gulp but so far I don't need its fancy streaming and plugins. So I searched for something simpler, and came across a comment that said something to the effect "if you're from those days, you could use make". They likely meant it in a derogatory way, but it made sense to me. I already have make on Windows and OS X and know how to use it. And it's dead simple to use it to simply run a few commands.

So now I can run "make bundle" or "make test". It's perfect for what I need. The hip kids can shake their heads all they want.

Thursday, August 25, 2016

Modules Again

Things had been running smoothly in suneido.js with the UMD approach. Until today when I went to run my code in the browser (I hadn’t done this for a little while because I’d been working on transpiling) and I got errors saying one of my modules couldn’t be found. I realized that I had started using a Node.js module in my code, which of course couldn’t be found in the browser. Crap!

I searched the web and most recommendations were to use browserify. It sounded straightforward, and it made sense to combine all my JavaScript into one file. So I installed browserify and switched my tsconfig.json back to commonjs.

But it wouldn’t work. It kept telling me it couldn’t find one of my modules. But as far as I could tell there was nothing different about that particular module or where it was imported.

I finally remembered that the flashy dependency graph feature of Alm had shown I had a circular dependency. Was that a problem with browserify? Searching the web didn’t find anything definitive, but there were some references to it being problematic.

It wasn’t hard to fix the dependency (and I’d been meaning to do it eventually anyway).

But it still didn’t work! Then I realized that the code shown in the Chrome developer tools debugger wasn’t my fixed version. Caching? A little more digging and I found you can right click on the refresh button and pick “Hard Refresh and Reset Caches”. Finally it worked! (Although I wondered if any of my other approaches would have worked if I’d done this???)

It seems like a reasonable solution, other than taking a dependency on yet another tool and adding another build step. (I do have to admit that npm makes it easy to install these tools.)

Friday, August 05, 2016

Alm TypeScript Editor

When I was reading TypeScript Deep Dive (recommended) I noticed a mention of an "alm" TypeScript editor. I'd never heard of it so I figured I'd better check it out.

The developer turned out to be the same as the author of the book - Basarat Ali Syed aka "bas". Who also turned out to be the the original developer of the atom-typescript plugin that I've been using.

Alm is a new project but it's moving quickly and is already full featured. And there's actually documentation :-) It has features like go to definition, find references, and even rename refactor. It also has some flashier features like dependency graphs and AST views. It has an outline side bar (alm calls it a Semantic View) which is a feature I really like in Eclipse and miss in Visual Studio. (We also have it in Suneido.)

The current "packaging" is a little quirky - you start it from a command prompt and the actually UI runs in Chrome. It would be nice to see it packaged with Electron, like Atom, but that's not critical.

You can use Chrome's Save To Desktop to open it as a "bare" window without the browser chrome, but you still have to start Alm first. No doubt there's a way to automate that. Or you can use "alm -o" which will open it in a tab in Chrome, and then use something like the Chrome extension Open as Popup.

I was interested to see that it was using the CodeMirror JavaScript code editor component which is what we have been using in the suneido.js project. But recently it changed to use the Monaco editor which was written for Visual Studio Code and recently released as a separate open source component. That makes sense because Monaco is written in TypeScript, and TypeScript was the original target language for Visual Studio Code.

Alm leverages the actual TypeScript compiler for source code analysis, which seems like the ideal approach.

I've only used it for a few days and I'm still learning my way around, but it looks like it will be my preferred editor for TypeScript.

Thursday, August 04, 2016

TypeScript Modules

TypeScript will translate ES6 (ES2015) modules into a variety of formats e.g. CommonJS for Node.js or AMD for require.js in browsers.

I found it a pain to have to switch my tsconfig.json back and forth to run my tests in Node.js or to run the actual code in the browser. Inevitably I'd forget to switch and/or recompile and it wouldn't work. I had to check one or the other version of tsconfig.json into Git, but that would confuse anyone who downloaded and tried to run the code.

I could probably set up a build process to generate both versions but I never got around to figuring that out.

I knew some code was written in a way that would work with both CommonJS and AMD and wondered why TypeScript didn't generate that format. Guess what, it does :-) TypeScript calls this output "UMD" or "isomorphic". (See TypeScript Modules under Code Generation)

    "compilerOptions": {
        "module": "umd",

The generated code is slightly larger since it basically contains both versions, but that's not a big deal.

Eventually (I hope) software will support ES6/2015 modules natively and you won't have to translate at all.

For backgrounds see What is AMD, CommonJS, and UMD?

Wednesday, August 03, 2016

TypeScript const enums

In my previous post I mentioned that if you make an enum const then you don't get the run time bidirectional mapping between numbers and names. It turns out that's not quite true.

If you turn on the preserveConstEnums compiler option (set it to true) then TypeScript still emits the mapping object, even though it still in-lines the enum values.

That seems like the best of both worlds. But I couldn't get it to work. If you try to use the mapping, you get:
import { Token } from "./tokens"

=> A const enum member can only be accessed using a string literal.
I searched online and found some suggestions to type assert to "any", but it didn't work.
import { Token } from "./tokens"
(Tokens as any)[token]

=> 'const' enums can only be used in property or index access expressions or the right hand side of an import declaration or export assignment.
So I gave up and didn't use const.

Then I came across a slightly different type assertion that worked!
import * as tokens from "./tokens"
(tokens as any).Tokens[token]
As the saying goes: "All problems in computer science can be solved by another level of indirection"

Note: As is recommended, I'm using the newer "as" form of type assertions instead of the older <type> form.

Thursday, July 14, 2016

Lexer in TypeScript

One of the standard things I do when I'm investigating a new language is to implement Suneido's lexical scanner. Not because this is necessarily the best test of a language. It's more that I'm usually evaluating the language in terms of how it would work to implement Suneido. I realize that's a very biased view :-)

I wasn't planning to do this with TypeScript / JavaScript because suneido.js isn't intended to be an "implementation" of Suneido. It's a transpiler that runs on jSuneido (using the jSuneido lexer and parser) that translates Suneido code to JavaScript. The only manually written TypeScript / JavaScript code is the runtime support library. (And that's partly a bootstrapping process. Once the transpiler is functional much of the support library could be written in Suneido code.)

But as I implement the runtime support library, obviously I need tests. And I'm getting tired of writing the same tests for each version of Suneido. (Or worse, different tests for each version.) That's why I started developing "portable tests" some time ago. So I decided I should implement the portable test framework in TypeScript. At which point I remembered that this required a Suneido lexical scanner.

So I decided to write yet another version. I took about a day, which is about what I expected. And some of that time is still learning TypeScript and JavaScript. I based it mostly on the C# version. The Go version is more recent but Go has more language differences.

Overall, I was pretty happy with how it went. I find the repetition of "this." a little ugly. In Suneido you can abbreviate as just .foo, which is a lot more compact and less "noise" in the code, while still making it clear it's an instance variable.

I find JavaScript's equality testing somewhat baroque, the recommendation is to use === most of the time, but that won't always work for strings (because of primitive strings and object strings). And then there's the quirks of -0 versus +0, and NaN.

ES6 arrow functions and the functional methods like map are nice. And spread / rest (...) and for-of are good. I only specified types on function parameters and returns, but this worked well and gave me very little grief.

I was pleasantly surprised to find that TypeScript enum's automatically create a reverse mapping from name to number. Although the gotcha is that if you make them const you don't get this because they are inlined. Using a simple numeric enum doesn't let me attach extra information for parsing, but I'm not planning to write a parser so that's not a problem. The other issue is debugging, where all you see is a number.

Removing const from my token enum exposed another issue. I was relying on Atom's auto-compile but it only compiles the file you are editing, not the other files that may depend on it. So I went back to running tsc -w in a separate terminal window. (Open question - does anyone know why tsc lists some of my files as relative paths and some as absolute? It's consistent about which files, except that gradually more are switching to absolute. It's not causing any problems, I just can't figure out why.)

Although it can mean shorter code, I'm not a fan of "truthy" and "falsey" values. It was a source of problems 30 years ago in C. Now I got caught by it with a function that returned a number or undefined. I expected undefined to be "falsey" but I forgot that zero is also "falsey". With Suneido I took a stricter approach that things like if and while only accept true or false and throw an exception for any other type of value.

Now that I have a lexical scanner I can implement the portable test framework, which also shouldn't take me too long.

Versions, in order of age. cSuneido and jSuneido are production code and more full featured than the other more experimental versions.