Escargot and Kangax

[Achie72]

I'm not sure, if this is one of the goals of the project, or totally off-scope, but there is an project named Kangax, with lists various javascript engines, and shows their compatibility with various ECMAScript standards:
https://kangax.github.io/compat-table/es6/
I ran a quick rundown on the ES6 testsuite subset (which is still one of the most wanted ES subset), and here is a summary for it:

TOTAL: 689
PASS: 645
FAIL: 44

Success: 93%

My questions are:

• Do we want to have an official upload in the Kangax project, to show Escargot in the list with all the other js engines? (Firefox, Chrome, Safari, duktape, JerryScript)
• Do we want to run ESnext and ESPlus testsuites as well?
• Do we want to add the testsuite(s) and/or a testrunner into this project.
• Do we want this at all?

[clover2123]

@Achie72 Thank you for your great suggetion.

• Official upload to Kangax project
  Actually we have plan to advertise escargot project through online JS engine communities. It would be great if escargot can be listed on the kangax. Could you guide us how to upload on kangax project?

• ESnext and ESplus
  What do you mean ESnext and ESplus exactly? If these are terms for the next version of ECMAScript standard (atm ES2020) we don't have any plan to support these features right now. We will begin to support new features when the next ECMAScript released (maybe this summer).

• Adding kangax testsuite
  We already runs many testsuites including test262. I'm not sure that we should add this testsuite. Could you explain briefly about kangax testsuite? For example, is this testsuite really famous or widely used for JS engine test? or it contains any performance test? Indeed we need performance test suite for ES6+ features because atm we only measure performance by old-fashioned TCs such as octane or sunspider. Do you know any the newest performance testsuite?

[Achie72]

Could you guide us how to upload on kangax project?
Well it's a bit tricky and tedious, but not really a difficult thing.

• A new test-runner script must be added based on the others.
• A new engine must be added to the enviroments.json
• The wanted testsuites need to be tested with the testrunner.
• Data-*js files must be updated by hand with the given results
• Run build.js with node, which will generate the corresponding HTML files.
• PR to the repository and it's gh-page branch.

What do you mean ESnext and ESplus exactly? If these are terms for the next version of ECMAScript standard (atm ES2020) we don't have any plan to support these features right now. We will begin to support new features when the next ECMAScript released (maybe this summer).

ESNext TC-s are contained in this file, and ES2016Plus (sorry i typod it in the original) tests are in this. For me ES2016Plus seems like exaclty that, things from ES6+. Sadly i'm not sure about the ESNext as there are many thing i'm not familiar with in it.

Could you explain briefly about kangax testsuite?

They are little, mostly few lines of codes testing features, and corner cases of them. Mostly no performance tests, just checking whether a features works correctly or not.
About the other test suites: i know about Kraken and Dromaeo from Mozzilla, but i only used it once like a year or two ago. If i recall it correctly it a heavy load testing thingy: https://wiki.mozilla.org/Kraken

[Achie72]

I've run some tests already, and have a draft version of a test runner, so if you guys are okay with it, i can finish the rest of the to-do-s and create some PR in the Kangax table repository.

The only thing needed is the testsuites we want to have in the report. There is:

• ES5
• ES6
• ES7
• ES2016plus
• ESIntl
• ESNext
• non-standard

If you take a look into these, which do we need and which not, i can just run and update the result files.

[clover2123]

@Achie72 Looks great!
How about adding ES6, ES7 and ES2016plus test suites first into this project?
After running these test suites and fixing some bugs, we may upload Escargot to the official Kangax project.
Thanks!

[clover2123]

It seems that ES7 was deleted or replaced by ES2016plus. It would be better to add ES6 and ES2016plus only. Thank you.

[Achie72]

Made a quick ES2016plus run:
132 tests executed: 67 success, 65 fail
Heavily failing tests here atm:

• async functions
• shared memory and atomics
• BigInt
• Optinal Chaining

The _onExit feature i'll mention in the end will help here too. Async's featrues are in progress as far as i know (based on recent PR-s), so that will fix over time, but i'm not sure if the last two i mentioned are even the scope for the current flow of the project, as those are not mention in Support Feature List. Considering all this, i think this is a nice result.

---

So i have one minor problem with adding those tests and not just a testrunnner into this project.
All of those tests are contained in the corresponding data-es*.js files as commented out exec: lines, such as this:

name: 'block-level function declaration',
  category: 'bindings',
  significance: 'small',
  note_id: 'block-level-function',
  note_html: 'Note that prior to ES6, it was <a href="http://wiki.ecmascript.org/doku.php?id=conventions:no_non_standard_strict_decls">recommended</a> that ES5 implementations forbid block-level declarations in strict mode.',
  spec: 'http://www.ecma-international.org/ecma-262/6.0/#sec-functiondeclarationinstantiation',
  exec: function () {/*
    'use strict';
    if (f() !== 1) return false;
    function f() { return 1; }
    {
      if (f() !== 2) return false;
      function f() { return 2; }
      if (f() !== 2) return false;
    }
    if (f() !== 1) return false;
    return true;
  */},
  res: {
    babel6corejs2: true,
    tr: true,
    closure: true,
    ie11: true,
    firefox2: false,
    firefox46: true,
    opera10_50: false,
    chrome21dev: chrome.experimental,
    chrome41: true,
    node0_12: "flagged",
    node4: true,
    xs6: true,
    safari10: true,
    duktape2_0: false,
    nashorn1_8: true,
    nashorn9: true,
    nashorn10: true,
    graalvm: true,
    jerryscript2_0: false,
    escargot2_0_0: false,
  }
},

and all these are in only one file, for one es* standard. I can extract these into their own .js files one-by-one but i think that is way too tedious, long, and maybe unnecessary. Even so if these tests update in the future. If we add a testrunner into the kangax project, one can just clone that down, and run it easily, to see results, as the runners are already prepared for this script storage solution.

My suggestions:
Copy those data files as well into the project and a testrunner.
or
Add a script into \tools, which will clone down the kangax repository, and launch an escargot based runner from there.

---

Sidenote:
The result can go up even further, we just need an _onExit for Escargot, on which i will start asap, and will put up a PR.

Also tell me if you need an output for ES6 and ES2016plus, and i will put them up on Gist and share the links.

[clover2123]

Currently, escargot passes only some part of ES2016plus tests but it will get better soon as many new features are going to be updated.
This is our specific development plan.

• async functions : in progress
• shared memory and atomics : do not support
• BigInt : will support (2020 feature)
• Optional Chaining : will support (2020 feature)

---

For test method,

1. Copy those data files as well into the project and a testrunner.
2. Add a script into \tools, which will clone down the kangax repository, and launch an escargot based runner from there.

IMO the 2nd appoach looks great for me. Is this way also easy to upload to the kangax project? If not, please choose one that is convenient for maintaining and updating.

---

BTW, what do you mean _onExit exactly? Is this a kangax-related feature that escargot should support? If so, then you can add this feature under ESCARGOT_ENABLE_TEST macro in the source of escargot.

---

Finally, how about adding kangax test run to the Jenkins CI, so we can find the kangax out at each CI result?

[Achie72]

IMO the 2nd appoach looks great for me. Is this way also easy to upload to the kangax project? If not, please choose one that is convenient for maintaining and updating.

I liked that idea as well, but i wanted to list all reasonable possibilities. I'm going for this one then.

To upload you just have to make a pull request with the updated html files correspoding to the testsuite you want to show Escargot in. They have a a builder script for that, so that shouldn't be that hard.

The harder part is to create the data files from which the builder creates the html ones. You have to insert every test result manually atm. I can create a script for that, but it's tricky and will take some time, as the data files are not that easily parse able (example data file) as you don't know how many engines are in the specific test's results and i think the only option there is to create a script counting braces {, and inserting escargot test results according to that. On top of that the test runner result printing has to be parsed as well, to find the corresponding section in the data file. It is do-able, but will take some bug testing and trial-error. (or we can push the Escargot results into the first row, but as we are adding new stuff to an old project, i don't think that is a proper solution)

BTW, what do you mean _onExit exactly? Is this a kangax-related feature that escargot should support? If so, then you can add this feature under ESCARGOT_ENABLE_TEST macro in the source of Escargot.

"With this option you can call a function after the user script and promises have ran, to be able to do assertions that are executed just before the process would exit." Which is used by many Promise tests in the testsuite (checking for async test passing). Example for this feature is already done in JerryScript

Finally, how about adding kangax test run to the Jenkins CI, so we can find the kangax out at each CI result?

I'm all for this idea, but i have two "major" concerns:

• If i add it now, each not fully implemented feature will fail, thus making the CI red, but if we turn them off, than it isn't much point in running them there. I'm open to any suggestion for this.
• The cloning of the kangax repository may add some time for the CI run

[clover2123]

If i add it now, each not fully implemented feature will fail, thus making the CI red, but if we turn them off, than it isn't much point in running them there. I'm open to any suggestion for this.
The cloning of the kangax repository may add some time for the CI run

For now, we may add the kangax testsuite to CI just to show the kangax result. After some bugs fixed, then we can enable the kangax on CI.
escargot already has several git modules for test, so cloning of the kangax repo would not affect the CI performance significantly IMO.

edit) If kangax repo is too huge to clone, we can alternatively add the kangax testsuite to js_vendor_tc repo