| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-08 | |||
| 17:17:34 | mriedem | ameeda: that's what i figured, so yeah just use that, we only need this dep for building docs | |
| 17:17:48 | stephenfin | dansmith: Anything that plays with HostState objects needs to call functions like this https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L32-L33 | |
| 17:18:08 | stvnoyes | Though I was using the volume as a data disk, not a boot disk. just to avoid the corruption on a m-a boot disk. | |
| 17:18:21 | ameeda | mriedem: cool :) | |
| 17:18:21 | stephenfin | That's because we set HostState.numa_topology to a value that we're retrieving from ComputeNode objects (ComputeNode.numa_topology) | |
| 17:18:39 | stephenfin | and, for legacy reasons, the ComputeNode.numa_topology attribute is a JSON blob | |
| 17:19:22 | stephenfin | dansmith: I've traced the flow here http://paste.openstack.org/show/666746/ | |
| 17:20:39 | stephenfin | dansmith: So, basically, if we want to _really_ get rid of all that crap, we need to bump the version of ComputeNode to 2.0 and store objects, or add some wrapper that will auto-convert the JSON to an object | |
| 17:20:55 | dansmith | stephenfin: that's going to be a lot of work | |
| 17:21:00 | stephenfin | Yup :( | |
| 17:21:11 | dansmith | definitely not happening as part of this effort before GA | |
| 17:21:41 | stephenfin | Anything that deals with converting the HostState objects from strings or non-object'y stuff _can_ go and the comments need to be updated as such | |
| 17:21:57 | stephenfin | Nope, definitely not. Far too much intertwined stuff here | |
| 17:23:59 | openstackgerrit | melanie witt proposed openstack/nova stable/ocata: Add release note for Aggregate[Core|Ram|Disk]Filter change https://review.openstack.org/542323 | |
| 17:25:00 | hrw | hm. whom to believe... "nova volume-attach dd9 uu-i-d" told "/dev/vdb" and on second try says "volume already attached" but horizon/instance/overview says "no volumes attached" | |
| 17:26:25 | hrw | multitail++ | |
| 17:27:37 | hrw | ok. logs show traceback. will keep unknown then | |
| 17:30:02 | bauzas | stephenfin: dansmith: I don't have the context, but yeah, it requires a major version bump to get rid of the past JSON string | |
| 17:31:15 | openstackgerrit | Ameed Ashour proposed openstack/nova master: Bindep does not catch missing libpcre3-dev on Ubuntu https://review.openstack.org/542340 | |
| 17:32:16 | openstackgerrit | Eric Fried proposed openstack/nova master: Avoid inventory DELETE API (no conflict detection) https://review.openstack.org/539712 | |
| 17:32:29 | efried | jaypipes: ^ another gory rebase | |
| 17:41:45 | sean-k-mooney | stephenfin: glad to hear. | |
| 17:42:14 | sean-k-mooney | stephenfin: did you modify our ovs-dpdk compilateion code to explcitly enable numa support? | |
| 17:44:54 | openstackgerrit | Marcin Juszkiewicz proposed openstack/nova master: Provide basic data for AArch64 support matrix/functionality https://review.openstack.org/541728 | |
| 17:45:21 | hrw | still several unknown states but should be closer to sanity | |
| 17:45:27 | hrw | mriedem: upgrade note added | |
| 17:46:16 | sean-k-mooney | stephenfin: try this before you get too far in grep CONFIG_RTE_LIBRTE_VHOST_NUMA /opt/stack/DPDK-*/build/.config. if it is set to n which i belive it will be the numa support in ovs/dpdk is disabled but that should allow you to get the worst case setup | |
| 17:48:06 | stephenfin | sean-k-mooney: I did not. Damn it | |
| 17:48:19 | sean-k-mooney | stephenfin: actullly wost case is nic on numa node 0, vcpu_pin_set allows only numa node 0 cpus and ovs_pmd_mask set pmd on numa node 1 | |
| 17:49:02 | sean-k-mooney | stephenfin: well if you want to mesure the maxium performance hit then leave it disabled and set up the topology above. | |
| 17:49:13 | mriedem | so apparently this is a regression unique to queens https://review.openstack.org/#/c/522161/ | |
| 17:49:20 | mriedem | recreated it here https://review.openstack.org/#/c/542232/ | |
| 17:49:26 | mriedem | doesn't fail in pike, ocata or newton | |
| 17:50:31 | sean-k-mooney | stephenfin: if you want to fix it just add another sed line here https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/libs/ovs-dpdk#L392-L399 | |
| 17:54:02 | openstackgerrit | Dan Smith proposed openstack/nova master: Bump compute RPC API to version 5.0 https://review.openstack.org/541005 | |
| 17:54:03 | openstackgerrit | Dan Smith proposed openstack/nova master: Compute RPC client bump to 5.0 https://review.openstack.org/541035 | |
| 17:57:44 | dansmith | melwitt: I was about to say this dep change proves your patch works, but.. I didn't think you could specify the url as the depends-on.. did I miss them adding that feature? https://review.openstack.org/#/c/540087/ | |
| 17:58:09 | melwitt | dansmith: yeah. you missed the controversy /me digs up link | |
| 17:58:18 | dansmith | ugh seriously? | |
| 17:59:10 | melwitt | here's the original post http://lists.openstack.org/pipermail/openstack-dev/2018-January/126535.html | |
| 17:59:16 | dansmith | yeah found it | |
| 17:59:19 | dansmith | me no likey | |
| 17:59:24 | melwitt | same | |
| 18:03:28 | dansmith | melwitt: mriedem: so on that change, we passed all but one in the last round, which was a volume timeout | |
| 18:03:34 | dansmith | same for the dep patch, | |
| 18:03:40 | dansmith | but on this recheck we failed a ton of volume things | |
| 18:04:03 | dansmith | I would expect these are unrelated, but... know of any known-broken volume stuff at the moment? | |
| 18:04:18 | openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | |
| 18:04:29 | mriedem | this job http://logs.openstack.org/87/540087/2/check/tempest-full-py3/0d74482/ ? | |
| 18:04:45 | dansmith | and tempest-full | |
| 18:06:24 | mriedem | looks like cinder-scheduler shit the bed http://logs.openstack.org/42/541442/3/check/tempest-full/3f33082/controller/logs/screen-c-sch.txt.gz?level=TRACE | |
| 18:06:27 | mriedem | which isn't our fault | |
| 18:06:38 | mriedem | Volume group "stack-volumes-lvmdriver-1" not found | |
| 18:07:24 | dansmith | yeah this is the other job: http://logs.openstack.org/42/541442/3/check/tempest-full/3f33082/controller/logs/screen-c-sch.txt.gz?level=TRACE | |
| 18:07:49 | openstackgerrit | Ameed Ashour proposed openstack/nova master: change instance_system_metadata column type https://review.openstack.org/526900 | |
| 18:15:19 | mriedem | dansmith: one lingering thing in https://review.openstack.org/#/c/541005/ | |
| 18:16:01 | openstackgerrit | Merged openstack/nova stable/pike: live-mig: keep disk device address same https://review.openstack.org/541642 | |
| 18:16:09 | openstackgerrit | Merged openstack/nova stable/pike: Fixes 'Not enough available memory' log message https://review.openstack.org/536297 | |
| 18:16:18 | openstackgerrit | Merged openstack/nova stable/pike: libvirt: use 'host-passthrough' as default on AArch64 https://review.openstack.org/532504 | |
| 18:16:31 | openstackgerrit | Merged openstack/nova stable/pike: Unmap compute nodes when deleting host mapping https://review.openstack.org/527560 | |
| 18:16:39 | openstackgerrit | Merged openstack/nova master: Add release note for Aggregate[Core|Ram|Disk]Filter change https://review.openstack.org/541018 | |
| 18:16:49 | dansmith | mriedem: replied | |
| 18:17:23 | mriedem | but you didn't remove the defaults from build_and_run_instance | |
| 18:17:25 | mriedem | in 5.0 | |
| 18:17:26 | mriedem | ... | |
| 18:18:28 | dansmith | mriedem: ah right, because I excluded that one unlike the others | |
| 18:18:35 | dansmith | so, fine, I'll remove it | |
| 18:18:47 | mriedem | there was another 5.x thing for migrate_data in rollback live migration | |
| 18:18:48 | mriedem | just noticed | |
| 18:19:26 | dansmith | ugh okay | |
| 18:21:08 | dansmith | mriedem: oh I can just remove that one | |
| 18:21:22 | dansmith | not sure what that was about, because it's already converted in live_migration always | |
| 18:22:17 | dansmith | we're even setting it on an object that enforces the type right before both calls to that | |
| 18:22:23 | dansmith | so we'd have been broken already | |
| 18:24:26 | mriedem | true | |
| 18:24:38 | mriedem | so yeah just remove those 2 things and i'm +2 | |
| 18:26:15 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: Marker reset option for nova-manage map_instances https://review.openstack.org/539501 | |
| 18:27:58 | sean-k-mooney | Jeffrey4l: hi are you still working on https://bugs.launchpad.net/nova/+bug/1414895 i recently deployed stable pike and have this issue for standard livemigrations not just host evacuate | |
| 18:27:59 | openstack | Launchpad bug 1414895 in OpenStack Compute (nova) "failed compute node didn't delete instance's path directory in init_host" [Low,In progress] - Assigned to Jeffrey Zhang (jeffrey4l) | |
| 18:28:10 | openstackgerrit | Stephen Finucane proposed openstack/nova master: WIP: Ensure 'HostState.numa_topology' is always an object https://review.openstack.org/542362 | |
| 18:28:20 | stephenfin | dansmith, bauzas: ^ | |
| 18:28:49 | stephenfin | It's rather low priority, but I would appreciate a sanity check on that at some future date | |
| 18:28:53 | stephenfin | Context here http://paste.openstack.org/show/666746/ | |
| 18:29:00 | sean-k-mooney | stephenfin: am could that break people who use the jason filter also do we care? | |
| 18:29:41 | stephenfin | sean-k-mooney: NUMATopologyFilter, I assume? | |
| 18:30:23 | Jeffrey4l | sean-k-mooney, no i am not working on it. feel free to take it. | |
| 18:30:58 | stephenfin | sean-k-mooney: It should not because we currently have code to handle both objects or strings https://github.com/openstack/nova/blob/master/nova/scheduler/filters/numa_topology_filter.py#L32-L33 | |
| 18:31:01 | sean-k-mooney | stephenfin: no the json filter allow you to match on any key/value pair on the hostsate dictionary | |
| 18:31:19 | stephenfin | ...which is what I want to delete | |
| 18:31:28 | stephenfin | sean-k-mooney: Hmm, I'm not sure | |
| 18:31:58 | sean-k-mooney | stephenfin: i think it might break it but for the second quest i also dont think we should really care | |
| 18:32:26 | sean-k-mooney | stephenfin: this filter to be specific https://github.com/openstack/nova/blob/master/nova/scheduler/filters/json_filter.py | |
| 18:32:53 | stephenfin | sean-k-mooney: I didn't even know that was a thing. I guess we could convert back to a JSON blob specifically for that string | |
| 18:33:00 | stephenfin | Or, more sanely, blacklist that key | |
| 18:33:03 | sean-k-mooney | stephenfin: you use it with scheduler hints | |
| 18:33:41 | stephenfin | I don't even think that would work at the moment. The 'numa_topology' string is one of those o.v.o object-in-a-string kind of strings | |
| 18:34:09 | stephenfin | i.e. with 'nova_object.version' and 'nova_object.changes' keys | |
| 18:34:25 | stephenfin | Probably a good point to note in the review | |
| 18:34:33 | stephenfin | And now, I'm off home o/ | |
| 18:34:35 | sean-k-mooney | Jeffrey4l: ok good to know. i dont know if i will have time to try and fix it but i hit it with kolla ansible deploy stable pike image for docker hub using external ceph | |