| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-04 | |||
| 13:27:52 | mriedem | it's something about the networks to pypi for those nodes | |
| 13:28:07 | cdent | that's...bizarre? | |
| 13:28:47 | frickler | https://review.openstack.org/#/c/649096/ should help with that. if we can get it to pass all those unstable tests, that is | |
| 13:30:02 | cdent | mriedem: is there anything I need to do here, or is the situation in hand? | |
| 13:32:58 | mriedem | cdent: if the underlying issue is the same bug that ^ is fixing then i think we're ok | |
| 13:33:24 | efried | mriedem: if we're trying to cut RC2 today, we need to resolve https://review.openstack.org/#/c/649775/ ASAP, right? | |
| 13:33:27 | efried | What's the next step there? | |
| 13:34:01 | mriedem | efried: i'd like input from you and dansmith on that one, seems gibi thinks bumping the major and dropping the fields is better (alternative 1) and i tend to agree | |
| 13:34:05 | mriedem | i wrote that patch before bed in haste | |
| 13:34:23 | mriedem | taking my kid to the bus stop | |
| 13:34:53 | efried | mriedem: I've only skimmed, but was there a concern that the malicious user would be able to opt into the insecure version? and how was that allayed? | |
| 13:35:09 | efried | I'll read it more deeply here... | |
| 13:35:42 | dansmith | those fields have no place in a notification, best I can tell | |
| 13:38:38 | efried | dansmith: I think that's agreed; the question is just whether we can get away with removing them from the existing version since nobody's using it yet, or if we need a new version | |
| 13:38:50 | efried | in which case can we still get hacked by people asking for the old one | |
| 13:39:35 | dansmith | the *only* reason to do the 1.1 version is if we really want to backport that or something | |
| 13:40:12 | dansmith | but since, as matt said on the review, the user doesn't get to choose what they get, I think it'd be better to signal "no, these really aren't going to be in your notification" with a major version bump | |
| 13:40:58 | dansmith | what I meant by "want to backport" is "want to backport without the appearance of a major bump and breaking change" | |
| 13:41:02 | efried | there's not like a way to spoof a client asking for the old version? | |
| 13:41:03 | dansmith | but I don't think we should do that | |
| 13:41:17 | dansmith | you don't ask not notification versions | |
| 13:41:20 | dansmith | you get what you get | |
| 13:41:27 | dansmith | the version is just a signal to tell you something has changed | |
| 13:41:36 | efried | okay, neat. Then I'm on board. | |
| 13:41:40 | dansmith | it's not like objects where we try to backport a version to the one you want | |
| 13:43:00 | efried | cool | |
| 13:45:45 | mriedem | ok i'll bump the major and drop the fields | |
| 13:45:54 | mriedem | do we still by default emit both legacy and versioned.... | |
| 13:46:06 | mriedem | yup | |
| 13:46:12 | mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#notifications.notification_format | |
| 13:46:37 | mriedem | the amount of object translation work to build these payloads is likely a detriment to scheduling performance as wel | |
| 13:46:42 | mriedem | *well | |
| 13:46:58 | mriedem | anywho, that's a different known issue | |
| 13:49:25 | dansmith | mriedem: the base object has some version that increments with each subclass change? | |
| 13:49:49 | dansmith | oh, nm | |
| 13:49:52 | dansmith | event_type | |
| 13:49:56 | dansmith | which won't need to change with this | |
| 13:55:15 | efried | Nova meeting in five minutes in #openstack-meeting | |
| 14:03:13 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/649775 | |
| 14:03:23 | mriedem | gibi_off: stephenfin: dansmith: melwitt: efried: ^ | |
| 14:03:35 | stephenfin | ack | |
| 14:03:39 | efried | thx mriedem | |
| 14:16:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: devstack: Remove 'tempest-dsvm-tempest-xen-rc' https://review.openstack.org/650018 | |
| 14:22:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add docs on what not to include in notifications https://review.openstack.org/650020 | |
| 14:27:56 | sean-k-mooney | jaypipes: looking at adrianc responce https://review.openstack.org/#/c/620115/29/nova/compute/manager.py@7193 i think its a resonable argument but im also happy to create the new function and use the synchronized decorator. | |
| 14:29:13 | sean-k-mooney | ill leave the decission up to you. the current decorator cant be used as a context manager but i think that could be added in the future to oslo concurancy | |
| 14:30:26 | sean-k-mooney | i think we would jsut have to change this return to a yeild https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L328 | |
| 14:31:36 | sean-k-mooney | and add the contextlib.contextmanager decorator https://docs.python.org/3/library/contextlib.html#contextlib.contextmanager | |
| 14:32:32 | sean-k-mooney | its something i need to tink about but its a sepreate form this change | |
| 14:42:22 | mriedem | efried: i shared a link to those charts i mentioned, you can peruse them at your leisure and if you think they are fine as further reading (albeit targeted topics) i can make them public and share on the ML or something | |
| 14:47:00 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Remove CellMappingPayload database_connection and transport_url fields https://review.openstack.org/650043 | |
| 14:47:36 | efried | mriedem: thank you. | |
| 15:16:34 | mriedem | hmm, because of that recent devstack change to add a 2nd shared public network we're always getting this in n-cpu logs during tempest runs now | |
| 15:16:35 | mriedem | http://logs.openstack.org/58/637058/15/check/nova-next/f4e8140/logs/screen-n-cpu.txt.gz?level=TRACE#_Apr_04_00_07_20_968719 | |
| 15:16:56 | mriedem | seems that should not trace, nor be an error really - the user made a bad request essentially but we catch it in compute rather than the api | |
| 15:18:06 | mriedem | yeah it's just a 400 INFO response in the api http://logs.openstack.org/58/637058/15/check/nova-next/f4e8140/logs/screen-n-api.txt#_Apr_04_00_07_20_975380 | |
| 15:20:00 | mriedem | huh surprising we even get that far, the request is empty Body: {"interfaceAttachment": {}} | |
| 15:25:45 | mriedem | https://bugs.launchpad.net/nova/+bug/1823198 is anyone has opinions | |
| 15:25:46 | openstack | Launchpad bug 1823198 in OpenStack Compute (nova) "NetworkAmbiguous traceback in nova-compute logs even though it's a user error" [Low,Triaged] | |
| 15:25:48 | mriedem | *if | |
| 15:53:22 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.openstack.org/621061 | |
| 15:53:48 | efried | dansmith: You around to hit https://review.openstack.org/#/c/649775/ ? | |
| 15:54:08 | dansmith | I will, I'm just about to finish my two hours of morning calls | |
| 15:54:14 | efried | cool, thanks. | |
| 16:01:58 | ganso | alex_xu: thanks for the +W! =) | |
| 16:03:50 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Don't warn on network-vif-unplugged event during live migration https://review.openstack.org/650060 | |
| 16:25:23 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: prevent common kwargs from glance client failure https://review.openstack.org/650064 | |
| 16:25:34 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: prevent common kwargs from glance client failure https://review.openstack.org/650065 | |
| 17:12:42 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle Invalid exceptions as expected in attach_interface https://review.openstack.org/650077 | |
| 18:32:09 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Allow driver to properly unplug VIFs on destination on confirm resize https://review.openstack.org/643024 | |
| 18:32:09 | openstackgerrit | Adrian Chiris proposed openstack/nova master: Move get_pci_mapping_for_migration to MigrationContext https://review.openstack.org/643023 | |
| 18:33:16 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not log a warning about not using compute monitors https://review.openstack.org/650092 | |
| 18:33:17 | mriedem | jaypipes: i know you love compute monitors ^ | |
| 18:33:39 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Excluding%20nova.compute.monitors.cpu%20monitor%20virt_driver.%20Not%20in%20the%20list%20of%20enabled%20monitors%20(CONF.compute_monitors).%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d | |
| 18:49:46 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle PortLimitExceeded in POST /servers/{server_id}/os-interface https://review.openstack.org/650097 | |
| 18:52:40 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle PortLimitExceeded in POST /servers/{server_id}/os-interface https://review.openstack.org/650097 | |
| 19:08:26 | mriedem | melwitt: have you seen this? intermittent unit test failure http://logs.openstack.org/06/641806/7/check/openstack-tox-lower-constraints/1f1fce6/job-output.txt.gz#_2019-04-03_22_36_18_975096 | |
| 19:12:27 | mriedem | looks like this unit test starts up a conductor service using the ServiceFixture which calls self.addCleanup(self.service.kill) | |
| 19:13:56 | melwitt | no, haven't seen that before. I've seen the error before, I remember helping to fix a bug for that way back when | |
| 19:14:40 | melwitt | this one https://review.openstack.org/555093 | |
| 19:15:26 | melwitt | and in the fail you linked, we're getting it during service.destroy() this time | |
| 19:15:55 | melwitt | which must mean in that flow somewhere we have a reader transaction context nested outside a writer transaction context | |
| 19:18:08 | mriedem | yeah not sure, also doesn't really show up much in logstash | |
| 19:18:13 | mriedem | so i guess i'll recheck and assume it's a fluke | |
| 19:18:27 | melwitt | looking at the code, so far I don't see how the nesting could be happening | |
| 19:33:55 | mriedem | efried_mtg: these failures http://logs.openstack.org/13/621313/21/check/openstack-tox-py36/ba2c848/testr_results.html.gz definitely started spiking after 4/1 with https://github.com/openstack/nova/commit/e5269f2fbb773953266fccbfd9df3f75e351eeb9 | |
| 19:34:06 | mriedem | i wonder if removing kipsdist = True in tox.ini is contributing | |
| 19:34:17 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22sqlalchemy.exc.InterfaceError%3A%20(pymysql.err.InterfaceError)%20(0%2C%20'')%5C%22%20AND%20tags%3A%5C%22console%5C%22&from=7d | |
| 19:34:23 | mriedem | *skipsdist = True | |
| 19:34:58 | mriedem | and it's always on node provider: limestone-regionone | |
| 19:45:47 | mriedem | tox on the limestone node is 3.8.4 and on an ovh node that passes tox is 3.8.1 | |
| 19:45:52 | mriedem | pymysql is the same between them | |
| 19:47:00 | efried | mriedem: What can I do to help? | |
| 19:47:13 | mriedem | hold me | |
| 19:47:36 | mriedem | but not in a tap out kind of way | |
| 19:47:46 | mriedem | i'm reporting a new bug and will push an e-r query | |
| 19:48:20 | efried | should I push a change adding skipdist back and see what happens? | |
| 19:49:02 | mriedem | idk, i wasn't involved in that lower-constraints fix so i don't know if that will screw up the lower-constraints job, but i guess you'd find out | |
| 19:49:24 | efried | according to the commit message, cdent thought skipdist was "otherwise superfluous" | |
| 19:49:35 | efried | so hopefully that means it at least won't hurt anything. | |