Dreamwidth news: 26 June 2015

Jun. 26th, 2015 05:45 pm
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise posting in [site community profile] dw_news
Hello, Dreamwidth! Greetings from Portland, where Dreamwidth has assembled for this year's Open Source Bridge. (Which remains my favorite conference ever for how wonderfully welcoming and diverse it is.)

Behind the cut:

* A fond farewell
* HTTPS
* Email woes: mostly fixed
* Multiple sticky entries
* Rescreening screened comments when they're edited
* Other new features and tweaks
* Pretty pretty pictures

Friday 26 June 2015 )

*

That's it from us for another update! As always, if you're having problems with Dreamwidth, Support can help you; for notices of site problems and downtime, check the Twitter status page; if you've got an idea to make the site better, you can make a suggestion. (I'm still a lot behind on the suggestions queue, though, just as a warning.)

Comment notifications may be delayed for up to an hour or two, due to the high volume of notifications generated after an update is posted to [site community profile] dw_news. This was posted at 5:45PM PDT (see in your time zone). Please don't worry about delayed notifications until at least two hours after that.

Changelog Digest for Fri, Jun 26

Jun. 26th, 2015 01:15 am
kareila: Rosie the Riveter "We Can Do It!" with a DW swirl (dw)
[personal profile] kareila posting in [community profile] changelog_digest

[dw-free]

dd0df6a: Issue #1412: Captcha failing on Android mobile platform
JS issue: Also use onchange for browsers that don't support onkeyup.
78320ca: Issue #1419: New Update Page degrades weirdly without Javascript
Tweak font size/space when displaying fallback text.
efe2e88: Issue #1424: Cleanup Routing API endpoints.
Make lookup-routing work with API endpoints, and remove code duplication.
6be0ca9: Issue #1230: Screen comment edits when comments are screened
If a screened comment was unscreened, editing the comment rescreens.
c137415: Issue #1428: Profile BML is borked
Fix syntax error.
49fe438: Issue #1434: 3 themes for Crossroads by forests_of_fire
New themes: Enjoy the Love, Falling Leaves, Neon Sequins.
280e23e: Issue #1431: 2 new themes for Patsy by timeasmymeasure
New themes: Foggy Beach, Glossalia.
9745f30: Issue #1430: 6 new themes for Lefty by timeasmymeasure and forthwritten
New themes: Foggy Beach, Glossalia, All Flowers In Time, Beach Hut, +2 more.
8186ecf: Issue #1430: 6 new themes for Lefty by timeasmymeasure and forthwritten
Fix font capitalization and escaping.
f8d0a4b: Issue #1432: 7 new themes for Line Up by timeasmymeasure
New themes: Clear Heads, Crossroads, Deep Waters, Honeycomb, +3 more.
bc800e2: Issue #1006: Update links in site copy to point to HTTPS
Replace HTTP with HTTPS in hardcoded links in various places.
09c8f61: Issue #1006: Update links in site copy to point to HTTPS
Remove unused beta update page strings.
691682c: Issue #1006: Update links in site copy to point to HTTPS
Rename translation strings to allow nonfree changes to take effect.
290714c: Issue #1444: Adds background color to menu nav when we hover over its dropdown
New SCSS rule to add the background color when hovering over dropdown menu links.
09862fd: Issue #1433: 13 themes for Crisped by forests_of_fire
More new themes.
a74b5e9: Issue #1067: Tags label invisible in Planet Caravan/some themes
Add footer text color variable for Planet Caravan, and use it where needed.

[dw-nonfree]

558609c: Issue #1006: Update links in site copy to point to HTTPS
Replace HTTP with HTTPS in hardcoded links in various places.

Code push!

Jun. 25th, 2015 09:20 pm
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise posting in [site community profile] dw_maintenance
We will be performing tonight's code push in a few minutes. There'll be a brief downtime, but not longer than a few minutes. (Ideally, that is.)

I'll update this entry when we're back, and people can report issues here.

EDIT: New code is live! Please report any issues here.

We're currently having an issue with the image proxy for accessing the site via HTTPS -- images are currently failing to load. We'll have that fixed as soon as we can.

on configuration

Jun. 25th, 2015 03:06 pm
escapewindow: escape window (Default)
[personal profile] escapewindow

A few people have suggested I look at other packages for config solutions. I thought I'd record some of my thoughts on the matter. Let's look at requirements first.

Requirements

  1. Commandline argument support. When running scripts, it's much faster to specify some config via the commandline than always requiring a new config file for each config change.

  2. Default config value support. If a script assumes a value works for most cases, let's make it default, and allow for overriding those values in some way.

  3. Config file support. We need to be able to read in config from a file, and in some cases, several files. Some config values are either too long and unwieldy to pass via the commandline, and some config values contain characters that would be interpreted by the shell. Plus, the ability to use diff and version control on these files is invaluable.

  4. Multiple config file type support. json, yaml, etc.

  5. Adding the above three solutions together. The order should be: default config value -> config file -> commandline arguments. (The rightmost value of a configuration item wins.)

  6. Config definition and validation. Commandline options are constrained by the options that are defined, but config files can contain any number of arbitrary key/value pairs.

  7. The ability to add groups of commandline arguments together. Sometimes familes of scripts need a common set of commandline options, but also need the ability to add script-specific options. Sharing the common set allows for consistency.

  8. The ability to add config definitions together. Sometimes families of scripts need a common set of config items, but also need the ability to add script-specific config items.

  9. Locking and/or logging any changes to the config. Changing config during runtime can wreak havoc on the debugability of a script; locking or logging the config helps avoid or mitigate this.

  10. Python 3 support, and python 2.7 unicode support, preferably unicode-by-default.

  11. Standardized solution, preferably non-company and non-language specific.

  12. All-in-one solution, rather than having to use multiple solutions.

Packages and standards

argparse

Argparse is the standardized python commandline argument parser, which is why configman and scriptharness have wrapped it to add further functionality. Its main drawbacks are lack of config file support and limited validation.

  1. Commandline argument support: yes. That's what it's written for.

  2. Default config value support: yes, for commandline options.

  3. Config file support: no.

  4. multiple config file type support: no.

  5. Adding the above three solutions together: no. The default config value and the commandline arguments are placed in the same Namespace, and you have to use the parser.get_default() method to determine whether it's a default value or an explicitly set commandline option.

  6. Config definition and validation: limited. It only covers commandline option definition+validation, and there's the required flag but not a if foo is set, bar is required type validation. It's possible to roll your own, but that would be script-specific rather than part of the standard.

  7. Adding groups of commandline arguments together: yes. You can take multiple parsers and make them parent parsers of a child parser, if the parent parsers have specified add_help=False

  8. Adding config definitions together: limited, as above.

  9. The ability to lock/log changes to the config: no. argparse.Namespace will take changes silently.

  10. Python 3 + python 2.7 unicode support: yes.

  11. Standardized solution: yes, for python. No for other languages.

  12. All-in-one solution: no, for the above limitations.

configman

Configman is a tool written to deal with configuration in various forms, and adds the ability to transform configs from one type to another (e.g., commandline to ini file). It also adds the ability to block certain keys from being saved or output. Its argparse implementation is deeper than scriptharness' ConfigTemplate argparse abstraction.

Its main drawbacks for scriptharness usage appear to be lack of python 3 + py2-unicode-by-default support, and for being another non-standardized solution. I've given python3 porting two serious attempts, so far, and I've hit a wall on the dotdict __getattr__ hack working differently on python 3. My wip is here if someone else wants a stab at it.

  1. Commandline argument support: yes.

  2. Default config value support: yes.

  3. Config file support: yes.

  4. Multiple config file type support: yes.

  5. Adding the above three solutions together: not as far as I can tell, but since you're left with the ArgumentParser object, I imagine it'll be the same solution to wrap configman as argparse.

  6. Config definition and validation: yes.

  7. Adding groups of commandline arguments together: yes.

  8. Adding config definitions together: not sure, but seems plausible.

  9. The ability to lock/log changes to the config: no. configman.namespace.Namespace will take changes silently.

  10. Python 3 support: no. Python 2.7 unicode support: there are enough str() calls that it looks like unicode is a second class citizen at best.

  11. Standardized solution: no.

  12. All-in-one solution: no, for the above limitations.

docopt

Docopt simplifies the commandline argument definition and prettifies the help output. However, it's purely a commandline solution, and doesn't support adding groups of commandline options together, so it appears to be oriented towards relatively simple script configuration. It could potentially be added to json-schema definition and validation, as could the argparse-based commandline solutions, for an all-in-two solution. More on that below.

json-schema

This looks very promising for an overall config definition + validation schema. The main drawback, as far as I can see so far, is the lack of commandline argument support.

A commandline parser could generate a config object to validate against the schema. (Bonus points for writing a function to validate a parser against the schema before runtime.) However, this would require at least two definitions: one for the schema, one for the hopefully-compliant parser. Alternately, the schema could potentially be extended to support argparse settings for various items, at the expense of full standards compatiblity.

There's already a python jsonschema package.

  1. Commandline argument support: no.

  2. Default config value support: yes.

  3. Config file support: I don't think directly, but anything that can be converted to a dict can be validated.

  4. Multiple config file type support: no.

  5. Adding the above three solutions together: no.

  6. Config definition and validation: yes.

  7. Adding groups of commandline arguments together: no.

  8. Adding config definitions together: sure, you can add dicts together via update().

  9. The ability to lock/log changes to the config: no.

  10. Python 3 support: yes. Python 2.7 unicode support: I'd guess yes since it has python3 support.

  11. Standardized solution: yes, even cross-language.

  12. All-in-one solution: no, for the above limitations.

scriptharness 0.2.0 ConfigTemplate + LoggingDict or ReadOnlyDict

Scriptharness currently extends argparse and dict for its config. It checks off the most boxes in the requirements list currently. My biggest worry with the ConfigTemplate is that it isn't fully standardized, so people may be hesitant to port all of their configs to it.

An argparse/json-schema solution with enough glue code in between might be a good solution. I think ConfigTemplate is sufficiently close to that that adding jsonschema support shouldn't be too difficult, so I'm leaning in that direction right now. Configman has some nice behind the scenes and cross-file-type support, but the python3 and __getattr__ issues are currently blockers, and it seems like a lateral move in terms of standards.

An alternate solution may be BYOC. If the scriptharness Script takes a config object that you built from somewhere, and gives you tools that you can choose to use to build that config, that may allow for enough flexibility that people can use their preferred style of configuration in their scripts. The cost of that flexibility is familiarity between scriptharness scripts.

  1. Commandline argument support: yes.

  2. Default config value support: yes, both through argparse parsers and script initial_config.

  3. Config file support: yes. You can define multiple required config files, and multiple optional config files.

  4. Multiple config file type support: no. Mozharness had .py and .json. Scriptharness currently only supports json because I was a bit iffy about execfileing python again, and PyYAML doesn't always install cleanly everywhere. It's on the list to add more formats, though. We probably need at least one dynamic type of config file (e.g. python or yaml) or a config-file builder tool.

  5. Adding the above three solutions together: yes.

  6. Config definition and validation: yes.

  7. Adding groups of commandline arguments together: yes.

  8. Adding config definitions together: yes.

  9. The ability to lock/log changes to the config: yes. By default Scripts use LoggingDict that logs runtime changes; StrictScript uses a ReadOnlyDict (sams as mozharness) that prevents any changes after locking.

  10. Python 3 and python 2.7 unicode support: yes.

  11. Standardized solution: no. Extended/abstracted argparse + extended python dict.

  12. All-in-one solution: yes.

Corrections, additions, feedback?

As far as I can tell there is no perfect solution here. Thoughts?

Code push Thursday

Jun. 25th, 2015 01:48 am
denise: Image: Me, facing away from camera, on top of the Castel Sant'Angelo in Rome (Default)
[staff profile] denise posting in [site community profile] dw_maintenance
We are currently planning a code push for about 18 hours from now (the night of Thursday 25 June). We'll probably begin around 9PM Pacific time, although it might be a bit later. As always, we'll update here and our Twitter offsite status when we begin.

Changelog Digest for Wed, Jun 24

Jun. 24th, 2015 05:14 pm
kareila: "PERL!" (perl)
[personal profile] kareila posting in [community profile] changelog_digest

[dw-free]

abd5756: Issue #1332: add title text for the tagnav image
Add helpful title text to explain what the tagnav button does.
29b9a39: Issue #1344: Fix Google Docs spreadsheet embeds
Allow both "spreadsheet" and "spreadsheets" format of URLs to work.
2373e29: Issue #1407: siteskins.blueshift.alt contains typo
Fix a minor typo in the description of the Blueshift site skin.
7d8f012: Issue #1079: Investigate/remove dead code in views/multisearch.tt and cgi-bin/DW/Controller/Search/Multisearch.pm
The site search function no longer has a "nav" mode, so remove related code.
6ebd254: Issue #1398: Admin Console Reference missing CSS styling
Add CSS styling for descriptions of enabled vs. disabled commands.
576c52b: Issue #1398: Admin Console Reference missing CSS styling
Add accessible text indicating enabled vs. disabled console commands.
fbe709e: #1279: banning a user does not remove them from your profile
Remove banned users from the list of users to display on the profile page.
6a875d5: Issue #1046: Convert BML files under htdocs/support
Converts "Support High Scores" page to TT and Foundation.
fd27079: Issue #1400: Ban and Unban Accounts page undiscoverable
Add a link to /manage/banusers from /manage/circle/edit.
f1956ae: Issue #1415: Fix alignment issues in the entry actions
Tweak styling on the beta update page: fix alignment issues.
504a142: Issue #1415: Fix button border styling
Tweak styling on the beta update page: fix button border styling.
d2f3d49: Issue #1315: spamreports by IP does not recognize IPv6 IPs
Allow spamreports to handle IPv6 addresses.
d2ccdc3: Issue #1416: Remove explicit local cgi-bin inclusion in ljlib.pl
Prevent ext/local/cgi-bin from overriding modules in cgi-bin.
3c76da3: Issue #923: $r->connection->remote_ip removed on Apache 2.4
Change method name in conjunction with Apache upgrade.
c88a908: Issue #786: Feeds won't accept HTTPS URLs
Changing upstream useragent modules fixes this bug.
46be70f: Issue #786: Feeds won't accept HTTPS URLs
Change all instances of LWPx::ParanoidAgent to LWP::UserAgent::Paranoid.
4468441: Issue #1420: More fixes for overlaying cgi-bin
More changes to loading order and paths of included modules.
3c273bf: Issue #1417: Add shorter sitename options to username tag
Allow common shorthand in site attributes of user tags (e.g. site="lj")
f184f57: Issue #677: Use color_header_link properties for links in journal headers
Add separate header link color variables to all layouts.

scriptharness 0.2.0

Jun. 21st, 2015 02:13 am
escapewindow: escape window (Default)
[personal profile] escapewindow

I've been getting some good feedback about scriptharness 0.1.0; thank you. I listed the 0.2.0 highlights and changes in the 0.2.0 Release Notes, but wanted to mention a few things here.

First, mozharness' config had the flexibility of accepting any arbitrary key/value pairs from various sources (initial_config, commandline options, config files...). However, it wasn't always clear what each config variable was for, or if it was required, or if the config was valid. I filed bug 699343 back in 2011, but didn't know how to tackle it then. I believe I have the solution now, with ConfigTemplates.

Second, 0.1.0 was definitely lacking a run_command() and get_output_from_command() analogs. 0.2.0 has Command for just running+logging a command, ParsedCommand for parsing the output of a command, and Output for getting the output from a command, as well as run(), parse(), get_output(), and get_text_output() shortcut functions to instantiate the objects and run them for you. (Docs are here.) Each of these supports cross-platform output_timeouts and max_timeouts in both python 2.7 and python3, thanks to the multiprocessing module. As a bonus, I was able to add context line support to the ErrorLists for ParsedCommand. This was also a want since 2011.

I fleshed out some more documentation and populated the scriptharness issues with my todo list.

I think I know what I have in mind for 0.3.0, but feedback is definitely welcome!

June 2015

S M T W T F S
 123456
78910111213
14151617181920
21222324 252627
282930    

Most Popular Tags

Style Credit

Expand Cut Tags

No cut tags
Page generated Jun. 29th, 2015 11:07 pm
Powered by Dreamwidth Studios