| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-13 | |||
| 17:55:28 | stephenfin | yumapath: Looks like an oslo.db issue. You'd be better off asking about it on #openstack http://codesearch.openstack.org/?q=No%20sql_connection%20parameter%20is%20established&i=nope&files=&repos= | |
| 17:56:10 | yumapath | ok | |
| 17:56:14 | yumapath | thanks | |
| 18:03:04 | mriedem | actually that's probably a superconductor issue | |
| 18:03:06 | stephenfin | melwitt: This is odd. What's going on here? https://review.openstack.org/#/c/325381/16/nova/tests/unit/compute/test_compute_mgr.py@279 | |
| 18:04:52 | mriedem | stephenfin: i sent yumapath here https://github.com/openstack-dev/devstack/blob/master/stackrc#L80 | |
| 18:04:56 | mriedem | and here https://docs.openstack.org/nova/latest/user/cellsv2-layout.html | |
| 18:05:01 | mriedem | they are likely failing on a reschedule | |
| 18:06:08 | stephenfin | I just realized I'm not actually on that channel | |
| 18:06:28 | mriedem | i'm not either usually | |
| 18:06:42 | mriedem | unless someone comes here with a problem and we shoo them off and i think i have an answer | |
| 18:07:14 | stephenfin | Heh, fair. I figured it was a lack of a configuration option that oslo.db cared about | |
| 18:07:24 | mriedem | with devstack that's pretty hard to screw up | |
| 18:07:28 | mriedem | unless you messed with stuff | |
| 18:11:57 | frickler | mriedem: stephenfin: https://bugs.launchpad.net/nova/+bug/1733933 | |
| 18:11:58 | openstack | Launchpad bug 1733933 in OpenStack Compute (nova) "nova-conductor is masking error when rescheduling" [Undecided,Confirmed] | |
| 18:12:17 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-status check for ironic flavor migration https://review.openstack.org/527541 | |
| 18:12:19 | mriedem | dtantsur|afk: dansmith: edleafe: ^ now with tests and docsy things | |
| 18:14:19 | mriedem | frickler: replied in the bug | |
| 18:14:30 | mriedem | frickler: if you rely on reschedules, you need to modify devstack | |
| 18:17:14 | frickler | mriedem: I think that answers it only partially. even if the reschedule doesn't work, I want to see the error from n-cpu on the instance and not have it replaced by "No sql_connection parameter is established" | |
| 18:18:45 | mriedem | frickler: ok fair enough, i re-opened it | |
| 18:19:08 | mriedem | but now i'm off for my pre-holidays haircutting | |
| 18:58:50 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: libvirt: Allow multiple volume attachments https://review.openstack.org/267587 | |
| 18:58:51 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 19:11:40 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: libvirt: Allow multiple volume attachments https://review.openstack.org/267587 | |
| 19:11:41 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 19:27:39 | openstackgerrit | Merged openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707 | |
| 19:27:46 | openstackgerrit | Merged openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357 | |
| 19:27:52 | openstackgerrit | Merged openstack/nova stable/ocata: fix nova accepting invalid availability zone name with ':' https://review.openstack.org/509659 | |
| 19:41:16 | mriedem | TheJulia: would you happen to know if there is an ironic grenade job that runs with resource classes? | |
| 19:41:29 | mriedem | i'd like to run that against https://review.openstack.org/#/c/527541/ | |
| 19:42:08 | TheJulia | I don't believe so... checking | |
| 19:43:07 | mriedem | hmm, doesn't look like the ironic grenade job runs the nova-status command anyway | |
| 19:43:14 | mriedem | compare http://logs.openstack.org/41/527541/1/check/legacy-grenade-dsvm-neutron-multinode/51ed76a/logs/grenade.sh.txt.gz#_2017-12-13_04_12_18_910 | |
| 19:43:18 | mriedem | to http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm-multinode-multitenant/2db59eb/logs/grenade.sh.txt.gz | |
| 19:43:36 | mriedem | i'll push an ironic patch to test things out | |
| 19:46:35 | TheJulia | mriedem: looks like we do set/use the resource class | |
| 19:46:58 | TheJulia | but we also set the mem/cpu/etc with a value of 0 for the flavor | |
| 19:47:43 | mriedem | yeah that's fine | |
| 19:47:56 | mriedem | this check would only fail if the node.resource_class isn't set i think | |
| 19:48:08 | mriedem | otherwise we'll migrate that node.resource_class into the instance.flavor.extra_specs | |
| 19:49:04 | TheJulia | we will likely want to publish a release note on ironic to stress that it MUST be set then | |
| 19:49:20 | TheJulia | Well, we've done that before, but now we really really mean it | |
| 19:49:51 | mriedem | you don't have to do that until nova drops the migration code | |
| 19:50:05 | mriedem | and we can't do that until we have at least this nova-status check to tell people if they are done with the migration | |
| 19:50:13 | mriedem | but yeah this is to try and get the ball rolling | |
| 19:51:12 | mriedem | hmm | |
| 19:51:20 | mriedem | why doesn't this ironic multinode grenade job upgrade nova? | |
| 19:51:21 | mriedem | http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm-multinode-multitenant/2db59eb/logs/grenade.sh.summary.txt.gz | |
| 19:51:44 | mriedem | this other one does http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm/41ce7b3/logs/grenade.sh.summary.txt.gz so i guess that's the one i care about | |
| 19:52:03 | mriedem | ok here we go http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm/41ce7b3/logs/grenade.sh.txt.gz#_2017-12-11_22_49_15_166 | |
| 19:52:05 | mriedem | cool | |
| 19:52:29 | ildikov | mriedem: hi | |
| 19:52:30 | TheJulia | oh, holdover from the old multinode scenario | |
| 19:52:47 | ildikov | mriedem: just wanted to give a heads up that I did some fix up on the libvirt patch for multi-attach | |
| 19:52:49 | TheJulia | one node should be getting upgraded if I remember correctly | |
| 19:53:03 | ildikov | mriedem: I will move up the support-matrix changes in the chain | |
| 19:53:35 | ildikov | mriedem: if you could take a look on that patch and see whether it's on the right track that would be great and then I can look into the locking in a follow patch on top of that one | |
| 19:53:41 | ildikov | mriedem: tnx :) | |
| 20:27:57 | mriedem | cfriesen: what can you tell me about https://blueprints.launchpad.net/nova/+spec/numa-node-pinning ? | |
| 20:29:12 | mriedem | cfriesen: are you guys carrying something for that? | |
| 20:29:19 | mriedem | seems like nested resource providers could help here | |
| 20:48:51 | mriedem | cfriesen: our product team is also interested in the use case in ^ - i copied in what my response was to them in the whiteboard | |
| 20:49:06 | mriedem | sounds like red hat is also interested but i don't have membership in the bugzilla ACL | |
| 20:49:37 | mriedem | https://bugzilla.redhat.com/show_bug.cgi?id=1519621 | |
| 20:49:38 | openstack | mriedem: Error: Error getting bugzilla.redhat.com bug #1519621: NotPermitted | |
| 20:53:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505 | |
| 21:08:28 | openstackgerrit | Merged openstack/nova master: Deprecate file injection https://review.openstack.org/522027 | |
| 21:10:41 | mriedem | ooo eff yeah | |
| 21:12:05 | mriedem | yikes 500 from placement http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-sch.txt.gz#_Dec_13_17_07_40_977103 | |
| 21:13:48 | mriedem | wth, tracing req-c9d0d841-a466-4b8a-aea1-aae993781b4f we get a NoValidHost, | |
| 21:13:52 | mriedem | http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-super-cond.txt.gz#_Dec_13_17_07_41_007395 | |
| 21:14:13 | mriedem | but looking at that in the scheduler logs, we don't even hit the filters for that request | |
| 21:14:14 | mriedem | http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-sch.txt.gz#_Dec_13_17_07_40_529855 | |
| 21:14:56 | mriedem | right around the time that request is starting to run in the scheduler, another one blows up with a 500 response from placement | |
| 21:15:00 | mriedem | but those should be separate threads yeah? | |
| 21:15:06 | mriedem | edleafe: ^ you see anything suspicious in here? | |
| 21:16:30 | mriedem | so uh | |
| 21:16:31 | mriedem | http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-placement-api.txt.gz#_Dec_13_17_07_40_968321 | |
| 21:16:34 | mriedem | that doesn't look good | |
| 21:16:38 | mriedem | jay_tampa: ^ | |
| 21:20:57 | mriedem | sdague: you wanna just push this through? https://review.openstack.org/#/c/519458/ | |
| 21:21:04 | mriedem | it sucks not being able to e-r check placement logs | |
| 21:21:51 | mriedem | oh maybe that's not an issue anymore | |
| 21:22:25 | sdague | it shouldn't impact e-r | |
| 21:22:35 | sdague | oh, actually maybe it does | |
| 21:22:42 | sdague | otherwise you get DEBUG logs, which overload it | |
| 21:23:16 | mriedem | logstash appears to be finding these | |
| 21:23:23 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22DBDeadlock%5C%22%20AND%20tags%3A%5C%22screen-placement-api.txt%5C%22%20AND%20voting%3A1&from=7d | |
| 21:24:09 | mriedem | i think the logstash problem with placement logs is multiline | |
| 21:24:20 | mriedem | yeah, i can't do message:"DBDeadlock" AND message:"_trait_sync" | |
| 21:24:31 | mriedem | clarkb: can we get https://review.openstack.org/#/c/519458/ in so i can do multiline queries on placement logs? | |
| 21:25:03 | mriedem | or maybe ^ doesn't fix that either... | |
| 21:29:03 | clarkb | mriedem: the filename does have screen in it right? | |
| 21:29:09 | clarkb | mriedem: so thats a valid fix regardless? | |
| 21:29:30 | clarkb | mriedem: as for multiline queries I would guess that is more a failing of the logstash parsing rules particularly for multiline events | |
| 21:30:11 | mriedem | clarkb: i definitely don't get the nice log level filters on the placement logs today | |
| 21:30:26 | clarkb | mriedem: ya thats going to be the chagne you linked then | |
| 21:30:40 | mriedem | i'm unable to live without log level filters | |