The unexpected return of JavaScript for automation(scriptingosx.com)

73 pointsby goranmoominvor 5 Jahren28 comments

The unexpected return of JavaScript for automation

https://scriptingosx.com/2021/11/the-unexpected-return-of-javascript-for-automation/

12 comments

nishsvor 5 Jahren

Interestingly I learnt recently that macOS includes a ``jsc'' command, which provides a built-in JavaScript REPL and interpreter [0]. It is part of the JavaScriptCore framework. I can find it at:

  /System/Library/Frameworks/JavaScriptCore.framework/Versions/Current/Helpers/jsc

But a Stack Overflow comment says it moves around between releases, so you may want to find it first using:

  $ find /System/Library/Frameworks/JavaScriptCore.framework -iname jsc

It doesn't support web-specific APIs such as window. But it includes extras such as a print function and a readline function.

  $ cat greet.js
  print("What is your name?")
  const name = readline()
  print(`Hello, ${name}!`)
  $ jsc greet.js 
  What is your name?
  ns
  Hello, ns!
  $

Have fun :)

[0]: https://furbo.org/2021/08/25/jsc-my-new-best-friend/

jonsenvor 5 Jahren

https://trac.webkit.org/wiki/JSC

hippichvor 5 Jahren

Am I reading it correctly? `const name = readline()` is blocking? Or it is some kind of async/promise?

lwouisvor 5 Jahren

macOS APIs have poor documentation overall, but JXA is on another level. It's not even like Carbon APIs which are still necessary to use today, but which documentation was removed from Apple's website. At least here, you can find an archive somewhere. No, JXA was simply never documented.

You basically have the AppleScript docs, which themselves are under-documented, and you have to guess what JXA APIs exist based off it. My recommendation is to use Safari and play with the debugger at runtime. That way you can test and see which APIs are available by trial and error. You can also hack around and display JS objects internal variables/prototypes to get an idea of what things may be available.

I've had to dig deep into this for my FOSS app recently, and I'll like to share these 2 gems of very-hard-to-find information:

- How AppleEvents (AppleScript or JXA triggered events) permission works (https://www.felix-schwarz.org/blog/2018/08/new-apple-event-a...)

- (Bitter) testimony from someone involved at JXA's inception (https://news.ycombinator.com/item?id=21803874)

Hopefully this message saves someone hours of research~

wdavidwvor 5 Jahren

We have been deploying infrastructures composed of Hadoop and Kubernetes clusters in JavaScript for years with Nikita (https://nikita.js.org) and the developer experience is great. We released patches and features very fast, writing unit tests is easy and the all process is very flexible. Most of our customers impose the usage of Ansible to accomplish similar tasks, and we lose the simplicity, control and agility. To achieve expressiveness similar to YAML, we use CoffeeScript which looks and feel like YAML with the benefit of a programming language. No need to work around templating limitations, not speaking about configuration management (it is so complex to work around it that we have an article just on this topic in the pipeline).

bovermyervor 5 Jahren

I'm surprised that CoffeeScript is still in use anywhere.

Before my current job, I would have been shocked to see JavaScript used as the primary infrastructure language, as I was so used to Chef and Ansible being the only CM tools, and Python being the de facto standard infra automation language.

Now, though, using JS for infra is starting to grow on me. I can't see myself using CoffeeScript ever again, but tools like zx and Pulumi are very interesting.

wdavidwvor 5 Jahren

I still use it a lot, particularly in unit tests and configuration. Take the tests of the CSV package, for example https://github.com/adaltas/node-csv/blob/master/packages/csv..., once you get used to its syntax, it is easier to read than plain JS. For Nikita, TypeScript would be appealing for code completion. In term of type checking, it will be double usage since all arguments are checked at runtime with JSON schema.

eins1234vor 5 Jahren

One really cool little JS library I've been using for a bunch of desktop automation tasks lately is nut.js and the lower level libnut library it's implemented on top of:

https://github.com/nut-tree/nut.js

https://github.com/nut-tree/libnut

It provides a means to send user input (mouse movement/clicks and key presses) and read and react to changes in visual state (through screenshots), and works across Windows, Linux and MacOS. It automates at a much lower level of abstraction than the approaches mentioned in the article that script against programmatic APIs.

What I really like about this lower level approach is that you don't need anyone's permission to automate anything, since there's no programmatic API that the system owners has to provide for you and thus can limit or take away when it becomes inconvenient (or more likely they'd just never provide it in the first place).

Any task that can be accomplished though looking at stuff on the screen and clicking a mouse and pressing keys on a keyboard (i.e. what a real person would do to accomplish the same task) can be automated, and it's actually surprisingly easy and effective to do this with nut.js. What really helps is that OpenCV has become ridiculously good and ridiculously fast at matching/identifying objects from a screenshot, with latencies usually in the low double digit miliseconds, so latency-based flakiness isn't nearly as much of an issue as I remember it being in the old days. I've also played around with OCR with tesseract but haven't had as much success with it in terms of perf, and remember seeing latencies of several seconds for even recognizing a single word from a tiny pre-cropped screenshot containing only the word itself, so these days I end up just taking screenshots of text to identify with OpenCV instead.

The main tradeoff to this approach compared to automation through APIs is that because it works by simulating real user inputs, it's not very amenable to running in the background while a user is actively interacting with the same machine, so a separate machine or VM is often needed. That's an acceptable tradeoff for some use cases but complete deal breaker for others, so YMMV, but just wanted to bring this cool little tool to people's attention.

account-5vor 5 Jahren

I'm a big user of AutoHotKey on Windows. How does this compare? I've been looking for a good Linux alternative for Autohotkey but there isn't one.

eins1234vor 5 Jahren

I actually came from AutoHotKey as well, specifically for the cross platform support!

I've found the dev experience with nut.js to be worlds ahead of AutoHotKey as well. You get to use a real programming language with proper modules, data structures, first-class functions, asynchrony, and have access to a vast ecosystem of third party libraries and tooling.

Some Windows specific APIs are easier to work with on AHK due to the collection of built-in functions specifically tailored for automation, but everything AHK is still possible with nut.js since you're just writing a node.js script and have access to libraries like https://github.com/node-ffi-napi/node-ffi-napi that can call native system libraries, with a bit more work involved.

aasasdvor 5 Jahren

I don't know about PyObjC, but JXA is very awkward when coming from JS: it brings in conventions from Objective-C, and it's not explained anywhere what patterns I'm supposed to use. E.g. with collections, it seems that instead of just values, you get some very lively views that change while you're trying to use them and refuse to act the same on subsequent access—so the robust way to get values from a collection repeatedly is to obtain the collection each time anew (or convert it to a sane Array). You have to figure out by yourself how AppleScript is translated to JS—or, more realistically, by copying examples from the web. Either I didn't find the proper documentation, or it's pretty much nonexistent. As the article mentions, the Github repo is the primary knowledge compendium on how the hell one uses JXA.

It's pretty much impossible to casually write JXA scripts now and then, just knowing JS and consulting reference docs—instead, it seems that you'll have to learn arcane ObjC conventions from the 80s and keep translating those to JXA, without much help from Apple. Add to this that Swift/ObjC docs themselves seem terse, done with a hefty bit of ‘fuck off’ attitude, and are more of ‘here's what we have’ reference with too little of ‘here's how to do the thing you want’. IIRC Swift docs just drop pages on old versions of classes and functions, so when Apple keeps rearranging things again in new versions, and you inevitably have old examples from the web, you have to grope for how each change works, instead of just opening the deprecated page and reading there that this new thing is now used.

So you have to translate JS ← JXA ← AppleScript ← Swift/ObjC, with resistance on every step.

Oh, and the system settings? I haven't seen comprehensive docs on them either. Just apple.stackexchange and a bunch of howto sites. I guess Apple prefers to provide living for these sites instead of spending some of the billions on a technical writer.

aasasdvor 5 Jahren

A, and by the way. Inevitably with a decent and widespread language you want to use third-party libraries, especially seeing as there are thousands upon thousands of them by now. Well, apparently you can load ‘compiled’ .scpt libraries—presumably done in either AppleScript or Javascript—but you can't just load a JS module from a file. There's a project that wraps JXA and uses Browserify to cram additional libraries down JXA's throat by merging them with the main script: https://github.com/johnelm/node-jxa

lewisjoevor 5 Jahren

I believe this is just the start. The more businesses move to SaaS models, automation takes a different shape as well, i.e automation scripts involving business objects within/outside the system. Most systems call them "custom scripts" - and the primary language of choice is Javascript. Why? Javascript engines have done a phenomenal job at sandboxing and extensions.

So that'd be another reason why JS will live long as a language of choice of automating cloud objects.

realrockervor 5 Jahren

> The more businesses move to SaaS models, automation takes a different shape as well, i.e automation scripts involving business objects within/outside the system.

Well put. I see this getting applied to cloud infrastructure too. Cloud providers are increasingly offering their products as pure software objects on which traditional software engineering principles can be applied. I have been experimenting with using typescript to define infra components for last 18 months and have seen higher code contributions from non-operators(most of them are typescript practitioners) w.r.t terrafrom or ansible.

[slightly shameless plug] Recently started using Deno + typescript and its even better. My thoughts on this here: https://adnaan.badr.in/blog/2021/11/20/deno-for-infrastructu...

andreineculauvor 5 Jahren

I can double down on that. I have developed a pattern to use any language (but we used js/python) to generate cloudformation templates. This has started in 2016, long before CDK. The idea originated in GCP's docs where they present both jinja and plain python as a way to generate iac.

I often got asked "what about terraform?" Yes, what about it? It is a +1 in our stack, lagged (maybe not now) behind cloudformation releases, it still doesn't give you the benefits of a proper programming language, let alone one that many are familiar with, and we will not have a multicloud problem in the foreseeable future. So yeah, what about it?

In retrospect, there is a negative. The industry is brainwashed so looking for a new job becomes: but do you know terraform? Completely overlooking whether one could simply learn it :) it ain't rocket science

onion2kvor 5 Jahren

and the primary language of choice is Javascript

The primary language for business automation is VBA - Visual Basic for Applications. There's probably a decent opportunity building something to convert businesses existing Office VBA scripts to JS that works in online tools.

jhgbvor 5 Jahren

Sounds like the big problem here is the sheer API size rather than the language.

pjmlpvor 5 Jahren

SaaS automation tools are just the second coming of Biztalk and BPEL tools.

Doing distributed computing since the late 90's feels like being part of groundhog day, every time a new stack comes along to solve all RPC problems.

Toutouxcvor 5 Jahren

I don't see how SaaS automation relates to JavaScript being used for automation in macOS..? Or is it just an association that popped up in your head?

onion2kvor 5 Jahren

You do get a lot of unexpected returns in JS.

bryanrasmussenvor 5 Jahren

might as well throw jsdb into the mix here http://www.jsdb.org/

manbartvor 5 Jahren

And Windows still comes with Jscript too.

mark_and_sweepvor 5 Jahren

Windows also runs Microsoft HTML Applications (HTAs) out of the box. These are useful if your script needs a GUI.

flohofwoevor 5 Jahren

Re: "Monterey has deprecated the pre-installed python on macOS"

Interestingly, there's a /usr/bin/python3 (3.8.9) on my new MBP which doesn't have the Apple deprecation message when starting into the REPL. Not sure if it was preinstalled or Xcode put it there though.

comexvor 5 Jahren

/usr/bin/python3 is technically preinstalled, but it’s just a stub that looks up the currently selected Xcode installation and execs the real python3 from there. (Same as /usr/bin/clang, /usr/bin/swift, and so on.)

tzsvor 5 Jahren

I've found JXA useful for simple web automation.

There is a site where I want to go to a few pages and extract some data. Their terms of service allow this as long as you do not access the site faster than a human browsing would do.

I used to do it with curl, with suitable delays between each page fetch.

Then they apparently got hit with some DOS attacks and started using Cloudflare protection and curl no longer worked.

I then switched to Selenium. That got stuck at hCaptcha, refusing to go on no matter how many times I correctly identified all the buses or trains or whatever.

Adding some experimental options in Selenium got around the hCaptcha loop:

  options.add_experimental_option("excludeSwitches", ["enable-automation"])
  options.add_experimental_option('useAutomationExtension', False)
  options.add_argument("--disable-blink-features=AutomationControlled")

A while later though the site started occasionally bringing up that Cloudflare "Checking your browser before accessing <site_name>" automatic check. That just sits there, occasionally reloading.

Same thing if I use undetected-chromedriver [1] which is supposed to be less likely to trigger anti-bot services.

Next up was using a bookmarklet. It was easy to make a bookmarklet that used an XMLHttpRequest to post the content of document.documentElement.innerHTML to a server of mine for processing. That bookmarklet could then advance to the next page of interest. It wouldn't be as nice as the prior solutions because I'd have to click the bookmarket for every page, but that wouldn't be too bad.

The bigger annoyance there is if my server isn't HTTPS the XMLHttpRequest is blocked because of mixed content. In Chrome you can tell it to ignore that on a per site basis, but it isn't persistent across browser launches. There are also issues if the server is on my LAN, which runs into the "not let JavaScript on a page from the internet access the LAN because it might be trying to hack your IoT" stuff.

I was just starting to look into browser extensions to see if that would be a viable approach. I don't know if (1) an extension can save to the local filesystem (which I'd prefer instead of having to do that XMLHttpRequest runaround), or (2) if an extension can carry out actions on multiple pages without user intervention (if I still would have to do the click per page I can just stick with the bookmarklet approach).

I was even contemplating trying to make a fork of Chromium or Firefox with built-in support for this stuff.

Then I remembered AppleScript, and that things that support AppleScript can also be controlled with JavaScript which is much much much less weird than AppleScript.

Here's some JXA code that saves current tab's source from Safari:

  var cur = Application.currentApplication()
  cur.includeStandardAdditions = true;
  var app = Application('Safari');
  app.includeStandardAdditions = true;

  function save_page(file_name)
  {
    let source = app.windows[0].currentTab.source()
    let path = Path(file_name)
    let f = cur.openForAccess(path, {writePermission: true})
    cur.setEof(f, {to: 0})
    cur.write(source, {to: f, startingAt: 0})
    cur.closeAccess(f)
  }

  save_page("/tmp/page.html")

Here are the changes to do that for Chrome (sort of...it doesn't quite get the full source):

  3c3
  < var app = Application('Safari');
  ---
  > var app = Application('Google Chrome');
  8c8,9
  <     let source = app.windows[0].currentTab.source()
  ---
  >     let ct = app.windows[0].activeTab
  >     let source = app.execute(ct, {javascript: "document.documentElement.innerHTML"})

Changing pages is also easy. For Safari,

  app.windows[0].currentTab.url = "..."

I don't remember what it is for Chrome, but it is similarly easy. And on both there is also the option of injecting JavaScript onto the page to do the navigation, if for some reason just setting the URL isn't good enough.

(No Firefox examples because Firefox has absolutely abysmal AppleScript support).

With JXA then I can write a handful of short command line utilities that can get the current URL, get the current page content, and navigate to a given URL and I've got everything I need.

(There might be a way to sort of make it work with Firefox. I believe (but am not certain) that with JXA I can tell the system send mouse-click events to anywhere I want in an application window. Firefox does provide enough AppleScript support to get the title of the current page. So maybe the bookmarklet approach, with JXA watching for the page changes and clicking the bookmarklet, would work).

[1] https://github.com/ultrafunkamsterdam/undetected-chromedrive...

wizzwizz4vor 5 Jahren

There's a middle ground between a bookmarklet and an extension: userscripts. https://www.greasespot.net/ https://wiki.greasespot.net/Main_Page

[deleted]vor 5 Jahren

a-dubvor 5 Jahren

sigh