Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-05-29
15:27:28 gibi efried: you don't have to rush as I hope there will be some neutron code up this week as well and that could allow us to have some integrated test as wel
15:28:10 efried ack
15:57:10 openstackgerrit Jan Gutter proposed openstack/os-vif master: Add multiqueue field to VIFGeneric https://review.openstack.org/570959
16:12:51 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: Avoid showing password in log https://review.openstack.org/561850
16:14:28 openstackgerrit Lee Yarwood proposed openstack/nova stable/pike: Avoid showing password in log https://review.openstack.org/561850
16:14:52 openstackgerrit Lee Yarwood proposed openstack/nova stable/ocata: Avoid showing password in log https://review.openstack.org/561851
16:25:17 mgagne mriedem: what's the best way for me to review and test the allocation healer? I'm still running Mitaka. My understanding is that I would need to run a more recent version?
16:29:18 mriedem mgagne: yeah probably at least pike, although i'm not sure if any of the internal interfaces i'm using in that CLI have changed since then
16:30:09 mriedem mgagne: could be possible to setup a small pike test environment with the caching scheduler, create some servers, resize one or two of them, and then run the heal_allocations CLI
16:30:36 mriedem i can also test that out if/when i get some free time (first day back after the summit)
16:31:24 mriedem gibi: want to skip the notifications meeting today? i'm still getting caught up from last week
16:31:32 mgagne mriedem: I suppose master would be best? I can try to setup a test environment in our infra. It would however be a non-trivial amount of effort as I'm not familiar with devstack and would probably do it "our way".
16:32:22 mriedem mgagne: sure whatever you're most comfortable with is best
16:32:57 mgagne mriedem: alright, I will attempt to squeeze time for that.
16:33:12 mriedem great, thanks
16:34:58 mgagne mriedem: btw, regarding cellsv1 -> cellsv2, a light bulb lighten up at the summit. I was previously under the impression that you *had* to run cellsv2 in ocata and that cellsv1 was removing/non-compatible, making a migration impossible due to me requiring support for multi-cells.
16:36:07 mriedem nope
16:36:14 mriedem so you're 4.5% happier now right? :)
16:36:28 mgagne 99% ? =)
16:36:50 mriedem mgagne: our cells v1 CI job is still cells v1 with a single cell1 for the nova db in the child cell and cell0
16:37:01 mriedem it's not multiple child cells, but still cells v1
16:37:04 mgagne all that time... ^^'
16:37:50 mriedem cool, glad it's more clear now
16:42:05 mriedem jmlowe: did you get my 2nd email about checking the allocation records for the compute nodes that are failing to live migrate?
16:42:51 jmlowe I did, turns out my allocations are a mess
16:42:56 gibi mriedem: besides what I wrote in the summary mail today on the ML I have nothing else for the meeting
16:43:01 mriedem gibi: ok
16:43:06 jmlowe I cleaned up the double allocations
16:43:20 jmlowe then I found allocations on the wrong host
16:43:22 gibi mriedem: so I will open the meeting to see if somebody new shows up but will close it in 5 minutes
16:43:34 mriedem jmlowe: hmm, ok. still weird that the problem didn't show up with normal server create.
16:43:38 jmlowe I really could use that new force create of allocations
16:44:05 mriedem jmlowe: the force option mentioned in my heal_allocations patch?
16:44:23 jmlowe I'm whipping up something to check nova location vs placement allocation location
16:44:43 jmlowe yeah that, my current workaround is to delete then live migrate
16:45:47 mriedem ok. wonder how everything got so messed up in the first place, but like i said in the email, there were bugs in the pike GA regarding properly cleaning up allocations when something failed - those have all been fixed in stable pike point releases, but you could have had cruft from the GA
16:46:27 jmlowe I figure I've got no chance of things working as expected unless all of the placement allocations are correct
16:48:03 jmlowe I did have a thought, we do lots of shelving and unshelving, any chance allocations are getting correctly updated during unshelve?
16:48:24 jaypipes jmlowe: you mean shelve_offloading?
16:48:39 jmlowe yes
16:48:43 jaypipes jmlowe: because I don't believe shelving (without offloading) affects allocations. could be wrong though.
16:49:01 jaypipes shelve is possibly one of the worst designed things in Nova... </grumbling>
16:49:05 mriedem when we shelve offload, the allocations for the node on which the instance is running will be deleted
16:49:42 mriedem https://github.com/openstack/nova/blob/stable/pike/nova/compute/manager.py#L4496
16:50:07 dansmith he said "without offloading"
16:50:11 mriedem when you unshelve, you go through the scheduler which will create allocations against the selected destination host
16:50:44 mriedem where did he say 'without offloading'?
16:51:00 jmlowe Really need it for our workflows, scientist does data analysis and runs for several weeks, shelves and goes off to write up results, submits paper and waits a couple of months to hear back from reviewers, unshelves and reruns based on reviewers comments
16:51:01 dansmith <jaypipes>jmlowe: because I don't believe shelving (without offloading) affects allocations. could be wrong though.
16:51:02 mriedem jay said that
16:51:11 mriedem by default shelve offloads immediately
16:51:15 mriedem and removes allocatoins
16:51:18 dansmith yeah, I thought you were contradicting jaypipes' comment
16:51:33 mriedem no, i thought jay was asking for clarification
16:51:34 dansmith jaypipes: I learned something about the importance of shelve this summit by the way
16:51:41 mriedem to which jmlowe said "yes" to shelve_offload
16:51:46 jmlowe that default behavior is why I tend to drop the offloading when referring to it
16:51:51 dansmith jaypipes: related to my apparent misunderstanding of how AWS works
16:51:58 mriedem jmlowe: same here
16:52:15 mriedem jmlowe: so shelve / unshelve should be ok wrt allocations
16:52:44 mriedem https://review.openstack.org/#/c/494632/
16:52:57 mriedem that was in the pike GA also
16:53:01 jmlowe I'll start spot checking history once I get a list of all the errant allocations
16:53:52 cfriesen dansmith: I'll bite...what did you learn about shelve?
16:54:14 dansmith cfriesen: weren't you in that cyborg session?
16:54:51 cfriesen dansmith: if I was I missed the important bit. :)
16:55:34 dansmith cfriesen: that the AWS equivalent of our stop/start involves a reschedule and is actually a little more equivalent to our shelve(-offload) and unshelve in terms of freeing resources (and the associated billing)
16:56:19 cfriesen dansmith: ah, okay. makes sense. I never could figure out the logic of why we have stop/suspend/pause when they all still consume resources.
16:57:03 dansmith well, just lifecycle of a pet, but I had assumed we imported that from AWS for parity
16:57:58 jmlowe I'd kill for the ability to offline migrate a suspended instance btw
16:58:27 cfriesen jmlowe: should be technically doable.
16:58:55 jmlowe cfriesen: right, just move the definition and the file where the state is written?
16:59:34 cfriesen jmlowe: essentially. the resource tracking and wiring up the networking and storage and whatnot are the tricky bits
16:59:57 dansmith yeah, a little more complicated than that
16:59:58 dansmith I thought that cold migrate could do it though
17:00:31 cfriesen dansmith: if not, I bet it wouldn't be horrendous to add it
17:00:38 jmlowe haven't tried for a while
17:01:01 dansmith or do you mean actually suspended (i.e. memory state saved) ?
17:01:44 cfriesen dansmith: looks like we only cold-migrate "active" or "stopped" instances
17:02:00 dansmith yeah, I assumed he meant suspended==stopped
17:02:15 dansmith I'm not sure I understand the rationale for suspend in the cloud either
17:02:30 dansmith maybe faster boot for slow-to-start applications or something, but..
17:02:39 yumapath hi can i get some help on putting pci devices in passthrough mode
17:02:48 yumapath https://docs.openstack.org/nova/pike/admin/pci-passthrough.html
17:02:52 cfriesen dansmith: in order to manage disk space on compute nodes we ended up making "suspend" behave like "pause"
17:03:04 yumapath i followed this link , but not able to get things working
17:03:26 yumapath if anyone out there can help me that would be great
17:05:14 dansmith cfriesen: for what gain?
17:05:21 dansmith oh,
17:05:23 dansmith suspend like pause
17:05:29 dansmith so they don't take up extra space with the memory image?
17:12:31 jmlowe hmm, so it would seem out of 946 instances two were in the wrong place, about two dozen don't have allocations
17:14:14 jmlowe last instance action for the one in the wrong place was an errored live migration
17:15:06 openstackgerrit Merged openstack/nova master: Adding NVMEoF for libvirt driver https://review.openstack.org/482640
17:20:34 cfriesen dansmith: was away from keyboard. yeah, otherwise we'd need to reserve as much disk space as we have guest RAM, which might be an issue on small-disk compute nodes.
17:21:03 dansmith yeah, makes sense. unfortunately it changes the behavior the api user expects
17:23:44 cfriesen dansmith: agreed. though the practical difference between the two is minimal
17:24:28 jaypipes jmlowe: sorry, had a phone call... ok, sorry, hope you are ready for a stream of opinions coming from me... here goes...
17:24:49 dansmith cfriesen: well, if you're using pause for fast HA, the startup time may differ significantly, but also suspend should survive host reboots, power outages, etc
17:24:53 dansmith cfriesen: but yes, similar

Earlier   Later