Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-05-21
14:59:54 mriedem which makes debugging not so nice
15:04:54 mriedem the RequestContext.from_dict(context.to_dict()) stuff must overwrite the thread local context
15:05:08 mriedem which messes up the logging, even though those copies should be using the same request id
15:06:36 dansmith mriedem: is it just that the threads we spawn to do the scatter don't get the thread-local set to their new context?
15:10:14 mriedem RequestContext __init__ defaults overwrite=True so i'd think each one is updated
15:10:48 mriedem i'm tracing a cold migration in these logs http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/controller/logs/screen-n-super-cond.txt.gz#_May_16_19_46_23_815726 with req-bde576dc-5de2-4113-982d-025c65b97e8d which is the request starting in the API
15:11:01 mriedem but after that we hit the scatter/gather and lose the request id
15:11:28 mriedem after that it's using req-66d8138a-530a-4dec-b1cd-9c22e4402dd1
15:19:48 aspiers efried: yup thanks
15:31:38 kashyap` efried: This was re-proposed (despite me abandoning it): https://review.opendev.org/#/c/660271/ ("Re-propose the spec to allow specifying a list of CPU models")
15:31:52 efried I noticed that
15:31:55 efried haven't looked yet
15:32:09 kashyap` efried: Don't. After technical discussions w/ the libvirt / QEMU CPU modelling maintainers, we've concluded it's not a good idea
15:32:14 kashyap` At least not in its present form.
15:32:33 efried kashyap`: lmk if you want me to -2 bomb the sucker.
15:32:41 kashyap` efried: Hehe, please do.
15:32:48 mriedem cfriesen: ^
15:33:00 efried kashyap`: save me finding the one you originally abandoned, got it at your fingertips?
15:33:08 mriedem unrelated, it seems odd that we'd fail to stop a libvirt cirros guest after a minute http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/screen-n-cpu.txt.gz#_May_16_19_47_26_017509
15:33:10 kashyap` (It was approved previously during Stein-time; but I didn't pay sufficient attention to it.)
15:33:12 efried nm, see the link in the comment.
15:33:48 kashyap` mriedem: Noticed it only today?
15:34:35 mriedem noticed yes, but http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Instance%20failed%20to%20shutdown%20in%2060%20seconds.%5C%22%20AND%20tags%3A%5C%22screen-n-cpu.txt%5C%22&from=7d
15:34:40 efried kashyap`: I did one better, I abandoned it.
15:35:24 mriedem if we're burning up to a minute waiting to gracefully kill cirros guests in the gate that could explain in part why some jobs/tests are really slow
15:35:43 kashyap` efried: Thanks!
15:38:51 kashyap` mriedem: "Needs more investigation." (I'm on mobile data in the library; will check tomm early.)
15:40:44 mdbooth efried: https://review.opendev.org/#/admin/groups/540,members Everybody on that list works for RH except Claudiu, John, and Matt. I don't know Claudiu, and I understand mriedem has some administrative difficulties right now :/
15:41:21 mdbooth efried: So the project is entirely dependent on johnthetubaguy to land stable patches.
15:41:28 mriedem no it's not
15:41:36 mriedem https://review.opendev.org/#/admin/groups/530,members
15:41:42 mriedem there is an #openstack-stable channel
15:41:49 mriedem and people like mtreinish
15:42:09 mriedem i'd also say smcginnis but he's in the same boat as me
15:42:10 kashyap` mriedem: Aren't you referring to release managers group?
15:42:17 mdbooth Ok. John and Matt Treinish
15:42:18 kashyap` I think mdbooth is referring to "stable-core"
15:42:34 mriedem nova-stable-maint's parent group is stable-maint-core 
15:42:44 mdbooth kashyap`: Yeah, mriedem was referring to stable-maint-core, who I understand can ack everything
15:43:03 mriedem Alan Pevecalan.pevec@redhat.comChuck Shortchucks@redhat.comFlavio Percocofpercoco@redhat.com so much for that
15:43:16 mdbooth mriedem: Yeah, it's a bit RH loaded :/
15:43:20 mriedem at some point can RH just extend me a job offer and i can switch sides here and start approving all RH stuff?
15:43:47 mdbooth mriedem: You won't be allowed under trifecta rules ;)
15:44:06 cfriesen kashyap: what was the objection by the libvirt/qemu folks? As you recall, the weakness in the current model is that you can only specify a model on the node, and it affects all guests on that node.
15:44:12 aspiers mriedem: not if another Linux distro nabs you first ;-)
15:44:32 mriedem so pressure the foundation to figure out wtf is going on with huawei developers to be able to work on open source
15:45:04 kashyap` cfriesen: I don't know if that's a "weakness" per-se. But the objection was the we'd need more data; and it is finicky, error-prone.
15:45:43 kashyap` cfriesen: They also asked: Do you have data on why _not_ use the default 'host-model' -- that has the important double-advantage in that, it auto-adds the relevant flags
15:46:22 cfriesen kashyap`: host-model can be an issue if you happen to land on a brand new compute node that has more advanced hardware than most of the other nodes
15:46:31 cfriesen kashyap`: makes it hard to live migrate
15:47:01 kashyap` cfriesen: Not quite -- 'host-model' explicitly makes sure that you _can_ live-migrate across disparate hosts. That's one of the reasons why Nova defaults to it
15:47:03 sean-k-mooney cfriesen: yes that is true but its kind of a know issue
15:47:27 cfriesen sean-k-mooney: yeah, but the whole point of the CPU model feature was to fix that known issue. :)
15:48:03 sean-k-mooney kashyap`: what he is raising is if you have 10 skylake servers and 90 ivybridge servers then if you land on the skylake ones you will only be able to migrate to the skylake ones
15:48:21 cfriesen as it stands, I think you'd need to set up host aggregates for the different hardware generations, or something like that.
15:48:30 mriedem fwiw i've had a non-RH core nom in mind for stable
15:48:35 sean-k-mooney so you would set cpu_mode=custom cpu_mode=ivybridge to mitigate against that
15:48:46 mdbooth mriedem: Do say!
15:48:51 mriedem ill never tell!
15:49:06 mriedem i can start that conversation again with the nova stable maint team
16:01:05 sean-k-mooney cfriesen: that is one apprch yes all though if i ever get arround to it i really want to play with the idea of a boaring host weigher that use traits as a proxy for how intersting a host is and selects the one with the least traits
16:01:15 sean-k-mooney e.g. the most boaring hosts
16:01:44 cfriesen boaring? like wild pigs? :)
16:01:46 sean-k-mooney which woudl mean if you did not ask for a trait specificaly the weigher would prefer selecting the oldest host
16:01:56 cfriesen sean-k-mooney: yeah, that's not a bad idea
16:02:09 cfriesen me too
16:02:31 sean-k-mooney yes its a weigher that digs holes :)
16:03:19 cfriesen sean-k-mooney: you'd still have the problem though that if you happened to land on a "newer" node due to resource pressure you'd be stuck on that hardware for live migration
16:04:05 sean-k-mooney ya that is true but i acutlly want it for other reasons like avoiding my host that have gpus when you did not ask for them that sort of thing
16:04:05 cfriesen it just seems unnecessarily restrictive to say that all guests on a node must have the same virtual CPU
16:04:48 sean-k-mooney well that is what nova has always done to this point but ya im aware of the new spec too
16:05:27 sean-k-mooney personally i just use host-passthough and say screw it ill use host aggrates to seperate things if i need too
16:06:11 sean-k-mooney actully i used ot use AZs at intel for that and ould have one AZ per generation but that was just so i could select the generation simplely when lanuching a vm
16:06:13 mtreinish mriedem: do I need to review something?
16:06:37 mdbooth mtreinish: https://review.opendev.org/#/c/659317/ please?
16:15:29 sean-k-mooney efried: thanks for reviewing the video models stuff ill adress your commets after dinner
16:20:12 mriedem kashyap: https://bugs.launchpad.net/nova/+bug/1829896 for that guest not stopping thing, i checked the libvirtd logs and the power off requests all return with 0
16:20:13 openstack Launchpad bug 1829896 in OpenStack Compute (nova) "libvirt: "Instance failed to shutdown in 60 seconds." in the gate" [Undecided,New]
16:20:28 mriedem not urgent but is concerning
16:22:03 sean-k-mooney i wonder if the qemu monitor process had hung?
16:22:21 sean-k-mooney i would expect a non 0 return in that case but i dont know if that is the case
16:41:07 ganso mriedem: hi Matt! Just a heads up in case you haven't seen it and have some free time to look at: https://review.opendev.org/#/c/659338 Thanks in advance! =)
16:41:38 kashyap sean-k-mooney: Yeah, I'd expect this to return non-zero here, in of a hung QEMU mithe case of hung QEMU: 2019-05-16 19:47:19.996+0000: 10502: debug : virThreadJobClear:121 : Thread 10502 (virNetServerHandleJob) finished job remoteDispatchDomainShutdown with ret=0
16:46:52 kashyap mriedem: It seems to fail with the "MigrationsAdmin" test (but your bug descriptino says: 'snapshot'), whose purpose is "Add os-migration tests for Nova v2 API" (added in 2014).
16:48:20 mriedem kashyap: if you look at the logstash results it's on pretty much everything
16:48:56 mriedem migrate_disk_and_power_off in a cold migration / resize will also power off the guest
16:49:11 mriedem which is where i noticed it while debugging a cold migration test today
16:52:25 kashyap Hmm. So it is the actual guest
16:53:16 kashyap Damned if I could put a finger on why. Let me ask the libvirt folks, if they can think of anything
16:53:45 kashyap Wish I could reproduce the issue. Damn sure it won't if I try locally
16:54:47 kashyap One dev asks:
16:54:47 kashyap < aw> kashyap: is it the VM or the guest OS? I've had fedora fail to shutdown in a timely way for all sorts of reasons, some that I still haven't figured out
16:55:14 mriedem don't know, but these test vms in the gate runs are just cirros images
16:55:19 kashyap (I've told it's CirrOS; and it's the VM, aka. QEMU process)
17:01:49 openstackgerrit Elod Illes proposed openstack/nova stable/ocata: Fix incompatible version handling in BuildRequest https://review.opendev.org/660456
17:02:00 kashyap Can anyone please remind, where is the guest boot log from the serial console here: http://logs.openstack.org/56/656656/12/check/nova-multi-cell/6a403ba/compute1/logs/
17:09:07 sean-k-mooney i think i just spent an hour debuging a unit test failure because teh unit tests in osc are wrong an i coppied them to create my own...
17:15:21 sean-k-mooney nope i am wrong it was 2 hours but now they pass
17:30:31 sean-k-mooney aspiers: mriedem i just updated https://review.opendev.org/#/c/643578/ if there are any other changes you would like let me know with a comment :)
17:30:44 aspiers sean-k-mooney: ok will take a look thanks!

Earlier   Later