| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-12-13 | |||
| 18:14:19 | mriedem | frickler: replied in the bug | |
| 18:14:30 | mriedem | frickler: if you rely on reschedules, you need to modify devstack | |
| 18:17:14 | frickler | mriedem: I think that answers it only partially. even if the reschedule doesn't work, I want to see the error from n-cpu on the instance and not have it replaced by "No sql_connection parameter is established" | |
| 18:18:45 | mriedem | frickler: ok fair enough, i re-opened it | |
| 18:19:08 | mriedem | but now i'm off for my pre-holidays haircutting | |
| 18:58:50 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: libvirt: Allow multiple volume attachments https://review.openstack.org/267587 | |
| 18:58:51 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 19:11:40 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: libvirt: Allow multiple volume attachments https://review.openstack.org/267587 | |
| 19:11:41 | openstackgerrit | Ildiko Vancsa proposed openstack/nova master: WIP: Allow multi-attach in compute api https://review.openstack.org/271047 | |
| 19:27:39 | openstackgerrit | Merged openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/516707 | |
| 19:27:46 | openstackgerrit | Merged openstack/nova master: Move the claim_resources method to scheduler utils https://review.openstack.org/511357 | |
| 19:27:52 | openstackgerrit | Merged openstack/nova stable/ocata: fix nova accepting invalid availability zone name with ':' https://review.openstack.org/509659 | |
| 19:41:16 | mriedem | TheJulia: would you happen to know if there is an ironic grenade job that runs with resource classes? | |
| 19:41:29 | mriedem | i'd like to run that against https://review.openstack.org/#/c/527541/ | |
| 19:42:08 | TheJulia | I don't believe so... checking | |
| 19:43:07 | mriedem | hmm, doesn't look like the ironic grenade job runs the nova-status command anyway | |
| 19:43:14 | mriedem | compare http://logs.openstack.org/41/527541/1/check/legacy-grenade-dsvm-neutron-multinode/51ed76a/logs/grenade.sh.txt.gz#_2017-12-13_04_12_18_910 | |
| 19:43:18 | mriedem | to http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm-multinode-multitenant/2db59eb/logs/grenade.sh.txt.gz | |
| 19:43:36 | mriedem | i'll push an ironic patch to test things out | |
| 19:46:35 | TheJulia | mriedem: looks like we do set/use the resource class | |
| 19:46:58 | TheJulia | but we also set the mem/cpu/etc with a value of 0 for the flavor | |
| 19:47:43 | mriedem | yeah that's fine | |
| 19:47:56 | mriedem | this check would only fail if the node.resource_class isn't set i think | |
| 19:48:08 | mriedem | otherwise we'll migrate that node.resource_class into the instance.flavor.extra_specs | |
| 19:49:04 | TheJulia | we will likely want to publish a release note on ironic to stress that it MUST be set then | |
| 19:49:20 | TheJulia | Well, we've done that before, but now we really really mean it | |
| 19:49:51 | mriedem | you don't have to do that until nova drops the migration code | |
| 19:50:05 | mriedem | and we can't do that until we have at least this nova-status check to tell people if they are done with the migration | |
| 19:50:13 | mriedem | but yeah this is to try and get the ball rolling | |
| 19:51:12 | mriedem | hmm | |
| 19:51:20 | mriedem | why doesn't this ironic multinode grenade job upgrade nova? | |
| 19:51:21 | mriedem | http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm-multinode-multitenant/2db59eb/logs/grenade.sh.summary.txt.gz | |
| 19:51:44 | mriedem | this other one does http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm/41ce7b3/logs/grenade.sh.summary.txt.gz so i guess that's the one i care about | |
| 19:52:03 | mriedem | ok here we go http://logs.openstack.org/75/527075/1/check/ironic-grenade-dsvm/41ce7b3/logs/grenade.sh.txt.gz#_2017-12-11_22_49_15_166 | |
| 19:52:05 | mriedem | cool | |
| 19:52:29 | ildikov | mriedem: hi | |
| 19:52:30 | TheJulia | oh, holdover from the old multinode scenario | |
| 19:52:47 | ildikov | mriedem: just wanted to give a heads up that I did some fix up on the libvirt patch for multi-attach | |
| 19:52:49 | TheJulia | one node should be getting upgraded if I remember correctly | |
| 19:53:03 | ildikov | mriedem: I will move up the support-matrix changes in the chain | |
| 19:53:35 | ildikov | mriedem: if you could take a look on that patch and see whether it's on the right track that would be great and then I can look into the locking in a follow patch on top of that one | |
| 19:53:41 | ildikov | mriedem: tnx :) | |
| 20:27:57 | mriedem | cfriesen: what can you tell me about https://blueprints.launchpad.net/nova/+spec/numa-node-pinning ? | |
| 20:29:12 | mriedem | cfriesen: are you guys carrying something for that? | |
| 20:29:19 | mriedem | seems like nested resource providers could help here | |
| 20:48:51 | mriedem | cfriesen: our product team is also interested in the use case in ^ - i copied in what my response was to them in the whiteboard | |
| 20:49:06 | mriedem | sounds like red hat is also interested but i don't have membership in the bugzilla ACL | |
| 20:49:37 | mriedem | https://bugzilla.redhat.com/show_bug.cgi?id=1519621 | |
| 20:49:38 | openstack | mriedem: Error: Error getting bugzilla.redhat.com bug #1519621: NotPermitted | |
| 20:53:15 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Update nova-status and docs for nova-compute requiring placement 1.14 https://review.openstack.org/526505 | |
| 21:08:28 | openstackgerrit | Merged openstack/nova master: Deprecate file injection https://review.openstack.org/522027 | |
| 21:10:41 | mriedem | ooo eff yeah | |
| 21:12:05 | mriedem | yikes 500 from placement http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-sch.txt.gz#_Dec_13_17_07_40_977103 | |
| 21:13:48 | mriedem | wth, tracing req-c9d0d841-a466-4b8a-aea1-aae993781b4f we get a NoValidHost, | |
| 21:13:52 | mriedem | http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-super-cond.txt.gz#_Dec_13_17_07_41_007395 | |
| 21:14:13 | mriedem | but looking at that in the scheduler logs, we don't even hit the filters for that request | |
| 21:14:14 | mriedem | http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-n-sch.txt.gz#_Dec_13_17_07_40_529855 | |
| 21:14:56 | mriedem | right around the time that request is starting to run in the scheduler, another one blows up with a 500 response from placement | |
| 21:15:00 | mriedem | but those should be separate threads yeah? | |
| 21:15:06 | mriedem | edleafe: ^ you see anything suspicious in here? | |
| 21:16:30 | mriedem | so uh | |
| 21:16:31 | mriedem | http://logs.openstack.org/64/527564/1/gate/legacy-tempest-dsvm-py35/7db2d64/logs/screen-placement-api.txt.gz#_Dec_13_17_07_40_968321 | |
| 21:16:34 | mriedem | that doesn't look good | |
| 21:16:38 | mriedem | jay_tampa: ^ | |
| 21:20:57 | mriedem | sdague: you wanna just push this through? https://review.openstack.org/#/c/519458/ | |
| 21:21:04 | mriedem | it sucks not being able to e-r check placement logs | |
| 21:21:51 | mriedem | oh maybe that's not an issue anymore | |
| 21:22:25 | sdague | it shouldn't impact e-r | |
| 21:22:35 | sdague | oh, actually maybe it does | |
| 21:22:42 | sdague | otherwise you get DEBUG logs, which overload it | |
| 21:23:16 | mriedem | logstash appears to be finding these | |
| 21:23:23 | mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22DBDeadlock%5C%22%20AND%20tags%3A%5C%22screen-placement-api.txt%5C%22%20AND%20voting%3A1&from=7d | |
| 21:24:09 | mriedem | i think the logstash problem with placement logs is multiline | |
| 21:24:20 | mriedem | yeah, i can't do message:"DBDeadlock" AND message:"_trait_sync" | |
| 21:24:31 | mriedem | clarkb: can we get https://review.openstack.org/#/c/519458/ in so i can do multiline queries on placement logs? | |
| 21:25:03 | mriedem | or maybe ^ doesn't fix that either... | |
| 21:29:03 | clarkb | mriedem: the filename does have screen in it right? | |
| 21:29:09 | clarkb | mriedem: so thats a valid fix regardless? | |
| 21:29:30 | clarkb | mriedem: as for multiline queries I would guess that is more a failing of the logstash parsing rules particularly for multiline events | |
| 21:30:11 | mriedem | clarkb: i definitely don't get the nice log level filters on the placement logs today | |
| 21:30:26 | clarkb | mriedem: ya thats going to be the chagne you linked then | |
| 21:30:40 | mriedem | i'm unable to live without log level filters | |
| 21:38:14 | edleafe | mriedem: no idea what's causing that error | |
| 21:39:09 | mriedem | edleafe: the dbdeadlock error in placement is causing the 500 in the scheduler logs, | |
| 21:39:15 | mriedem | i just can't figure out why that also aborts the other request | |
| 21:39:19 | mriedem | which should be on a different thread | |
| 21:39:29 | mriedem | https://bugs.launchpad.net/nova/+bug/1738083 | |
| 21:39:30 | openstack | Launchpad bug 1738083 in OpenStack Compute (nova) "DBDeadlock when when syncing traits in Placement during list_allocation_candidates" [High,Triaged] | |
| 21:39:30 | edleafe | mriedem: yeah, I meant the deadlock | |
| 21:39:35 | mriedem | traits sync | |
| 21:41:25 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI for traits (v1.6) https://review.openstack.org/514643 | |
| 21:41:26 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Resource class set (v1.7) https://review.openstack.org/514644 | |
| 21:41:26 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: Usages per project and user (v1.8, v1.9) https://review.openstack.org/514646 | |
| 21:41:27 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: CLI allocation candidates (v1.10) https://review.openstack.org/514647 | |
| 21:41:27 | openstackgerrit | Andrey Volkov proposed openstack/osc-placement master: [WIP] Get resource provider by uuid or name https://review.openstack.org/527791 | |
| 21:42:05 | mriedem | this reminds me, | |
| 21:42:14 | mriedem | we should run multiple scheduler workers (2) in our CI | |
| 21:42:18 | mriedem | because we said we can since pike | |
| 21:42:41 | edleafe | mriedem: since we're only expecting db_exc.DBDuplicateEntry, not a DBDeadlock | |
| 22:11:47 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova-status check for ironic flavor migration https://review.openstack.org/527541 | |