| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-06-17 | |||
| 14:30:23 | sean-k-mooney | we had a concept of allowing boot to put instancve into states other then active. e.g. the boot to shelved usecase | |
| 14:30:44 | sean-k-mooney | would something like that make sense | |
| 14:31:01 | sean-k-mooney | TheJulia: ^ im just not sure what state you would want the instance to be in Nova | |
| 14:31:03 | TheJulia | sean-k-mooney: possibly, I'll pass along the idea | |
| 14:31:13 | dansmith | sean-k-mooney: you mean "we discussed that thing once upon a time" :) | |
| 14:31:15 | TheJulia | sean-k-mooney: ideally active to nova's perception | |
| 14:31:32 | sean-k-mooney | dansmith: yes | |
| 14:32:03 | dansmith | sean-k-mooney: I think the goal here is to use nova's scheduler but not the nova-compute part, which would kinda be akin to "we provision empty VMs on the hypervisor and then you do what you want with them, | |
| 14:32:10 | sean-k-mooney | TheJulia: so rather then a trait if i was to propose something like this i would proably use an extra spec | |
| 14:32:16 | dansmith | which goes against the whole "don't manage our VMs out of band" goal | |
| 14:32:40 | sean-k-mooney | dansmith: right it would conflict with that | |
| 14:32:43 | dansmith | and it would make alllll the other operations you can do on an instance have to be disabled somehow | |
| 14:33:03 | sean-k-mooney | we did have some driver that wanted to prespawn some vms and reallocate one on a boot request in the past right | |
| 14:33:05 | dansmith | like, what does it mean for you to do a reboot or rebuild or anything else on an instance in that case | |
| 14:33:36 | dansmith | I dunno about that | |
| 14:33:54 | sean-k-mooney | TheJulia: do they just not want to do the provisioning e.g. would boot without and image be an alternitive | |
| 14:35:43 | sean-k-mooney | TheJulia: actully ignoring the schduling bit is there a reason they cant just deploy using ironics api | |
| 14:37:12 | sean-k-mooney | presumable ironic could claim the resouces in placment when you go to ironic directly so that nova/placement would know the host is in use | |
| 14:50:09 | TheJulia | sean-k-mooney: trying to focus on the discussion call I'm on right now, I'll answer afterwards | |
| 14:51:05 | sean-k-mooney | TheJulia: no worries im watching an internal call too | |
| 15:18:17 | melwitt | gmann: have you seen this gate bug before? "ls: cannot access 'uwsgi*': No such file or directory" https://zuul.opendev.org/t/openstack/build/5c1bfbb79c314b11b518a8af51441a00/log/logs/grenade.sh.txt#5354 | |
| 15:18:59 | gmann | melwitt: yeah that is ongoing fixes in devstack, and we have merged all the stable branch fix now | |
| 15:20:19 | dansmith | since monday I think | |
| 15:20:19 | melwitt | gmann: ah thanks, I missed that | |
| 15:20:41 | melwitt | I had a thing on stable (stein) fail zuul overnight | |
| 15:21:10 | gmann | yeah, | |
| 15:21:14 | melwitt | so I was wonderin | |
| 15:21:34 | gmann | but grenade multinode job is still issue, fixing in ussuri by backporting running zuulv3 native - https://review.opendev.org/#/c/735948/1 | |
| 15:22:51 | gmann | train and stein might need to make it n-v till we backport the grenade zuulv3 till stein. that is we are discussing in qa channel | |
| 15:25:39 | melwitt | ok I'm seeing neutron-grenade-multinode and nova-grenade-live-migration and grenade-py3 failing on stein | |
| 15:28:50 | melwitt | and then nova-live-migration is failing with "/opt/stack/new/devstack/lib/infra: line 32: virtualenv: command not found" on stein https://zuul.opendev.org/t/openstack/build/509288c2a75a49609c2f7f4b2e158a60/log/logs/devstack-subnodes-early.txt#2948 | |
| 15:33:01 | gmann | melwitt: yeah, that need devstack-gate fix which is not preferred so i am trying to backport the grenade zuulv3 till train. | |
| 15:45:09 | melwitt | gmann: ok, thanks. devstack-gate is branchless I assume? | |
| 15:45:27 | gmann | melwitt: yeah. | |
| 15:49:25 | melwitt | gmann: thanks. have you heard of the other failure before "/opt/stack/new/devstack/lib/infra: line 32: virtualenv: command not found"? for nova-live-migration | |
| 15:54:02 | gibi | melwitt: could this be due to http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015425.html ? | |
| 15:55:13 | melwitt | gibi: oh maybe, thanks. I missed that, was looking for "gate" in ML | |
| 15:55:19 | gmann | melwitt: that is same issue grenade multinode job facing | |
| 15:55:29 | gmann | and need fix in devstack-gate | |
| 15:56:04 | melwitt | oh, the error message was different so I didn't realize it was related | |
| 16:11:09 | TheJulia | dansmith: the goal is more, have the nova-compute driver ask ironic to hold the node, set appropriate values to represent the user has possesion of it and allow the user to do the needful. | |
| 16:11:41 | TheJulia | sean-k-mooney: the conundrum is the users expect to use nova, and currently get resources via nova today, and what they are doing is basically going and making internal invasive chagnes inside the deployed instance to re-install the instance using a kickstart | |
| 16:11:58 | dansmith | TheJulia: yeah that's the same as what I said, just with nova-compute being slightly involved in the enablement of the out-of-band activities | |
| 16:12:26 | TheJulia | Well, it may be logical to enable setting the "leasee" field in ironic's api anyway | |
| 16:12:39 | TheJulia | which would allow operators to optionally grant direct api rights | |
| 16:13:05 | TheJulia | I guess there could always be a trait that populates itself that just signals "don't actually deploy this machine, just allocate it" | |
| 16:13:15 | dansmith | that's akin to nova setting up a libvirt VM and then tolerating the user or admin mucking with the underlying VM XML | |
| 16:13:18 | TheJulia | and we could do that under the hood in ironic fairly easily. | |
| 16:13:21 | dansmith | which today we declare as "not okay" | |
| 16:15:12 | TheJulia | Well, we should still eventually store who asked for the thing and if we have community consensus behind a "delayed deployment" trigger, then that is totally up to the operators and their running policy to decide if they want to permit or not to meet their use cases | |
| 16:17:23 | sean-k-mooney | dansmith: for what its worth we had someone asking about pxe booting nova instance a few weeks ago | |
| 16:18:01 | sean-k-mooney | they were confused by the fact that if you enable the qemu boot menu via the image porperty you can select pxeboot as an option in the boot menu via the console | |
| 16:18:24 | sean-k-mooney | but could not figure out how to configure it which makes sense since we dont support you pxe booting the vms booted by nova | |
| 16:18:27 | dansmith | sean-k-mooney: you mean asking again | |
| 16:19:04 | sean-k-mooney | it came up on irc like a month ago i assuem it has come up before | |
| 16:19:09 | dansmith | many times | |
| 16:19:44 | dansmith | I dunno if there would be any ironic parallels here, but changing the image from inside the guest means that anything we do after that based on the image metadata will be wrong | |
| 16:20:06 | sean-k-mooney | i basically just said thats not supported sorry, it just a leaky abstraction | |
| 16:20:18 | dansmith | like attaching a device and using the correct device model, or migrating somewhere that looks at image metadata for OS properties, etc | |
| 16:20:26 | sean-k-mooney | dansmith: well so i have heard of people upgradeing form like 16.04 to 18.04 | |
| 16:20:37 | sean-k-mooney | but then you get into the pets vs cattle debate | |
| 16:20:55 | dansmith | which is wholly uncloudy, but is a lot less bad than going from 16.04 to Windows 10 | |
| 16:20:59 | dansmith | yup | |
| 16:21:46 | sean-k-mooney | ya updating the same disto is not really a problem different openrating systemsn entierely like linux to windows or esxi is a tolltal differnt matter | |
| 16:25:01 | sean-k-mooney | TheJulia: honestly it sound like they should update there kickstart playbook to delegate the os imagiang and provisioning to openstack/ironic and then just deal with boot straping the deployed hosts | |
| 16:26:13 | TheJulia | the case that is coming up at CERN is apparently they are deploying nodes with configurations that are impossible to articulate in images, so they are making the necessary major changes to get to their required end state after the fact | |
| 16:27:04 | sean-k-mooney | TheJulia: sure that makes sense but i dont know wny they would not just orcestrat the software payload via ansible and have a seperate role to deploy the orginal server | |
| 16:27:19 | sean-k-mooney | TheJulia: kind of like kaobe | |
| 16:27:27 | TheJulia | Really, ideally they should jsut be talking with ironic in some of these cases, but I guess they are trying to force their users into cloudy mindsets/patterns/behaviors | |
| 16:28:22 | dansmith | yup | |
| 16:28:33 | TheJulia | It is a mixed vm/baremetal environment, fwiw | |
| 16:29:14 | dansmith | what are the "node configurations that are impossible to articulate in images" ? | |
| 16:29:17 | TheJulia | And high level consistency seems to be the key they need, it is almost like power users just need that extra next step | |
| 16:29:22 | sean-k-mooney | ya that is not really an issue johnthetubaguy could proably tell you more about how they orcestrate the deployment of the Square Kilometer Array (SKA) | |
| 16:29:34 | TheJulia | multi-disk/raid/lvm/etc | |
| 16:29:51 | TheJulia | dansmith: this came out of the whole disk partitioning discussion getting resurrected | |
| 16:30:18 | dansmith | ack, well, addressing that makes more sense to me than just saying "short-circuit the bottom half of nova because it's easier" | |
| 16:30:54 | TheJulia | except, then get into the whole pets/cattle argument again and everyone tableflips and walks away | |
| 16:31:29 | sean-k-mooney | cern have a blog post on that but it seam sto be down the stack hpc one is up however https://www.stackhpc.com/software-raid-in-ironic.html | |
| 16:31:38 | dansmith | that's not a convincing argument to do the short-circuit :) | |
| 16:32:54 | TheJulia | dansmith: it is a solid reason people have downstream patches and are why they are not upgrading though, because their business processes need something along those lines. Anyway, if we enable them to short circuit in ironci, I think that is fine. Nova will believe it is happy. they can do whatever special config assertion as long as we populate leasee which we should be doing anyway. | |
| 16:33:53 | sean-k-mooney | TheJulia: there are ways to do per instance raid config today howere right | |
| 16:34:01 | TheJulia | which _should_ make the world a happier place. Dogs and cats could get married and all | |
| 16:34:51 | dansmith | whatever, it's just avoiding an abstraction because it's hard | |
| 16:34:52 | TheJulia | sean-k-mooney: it is not just raid sadly, the convergence seemed to be that there are teams with whole kickstart configs and to try and articulate that with traits will just make flavor/trait explosion even worse. | |
| 16:35:03 | dansmith | even though our only purpose in life is to be an abstraction | |
| 16:37:01 | TheJulia | in a perfect end state, I'd prefer them to get everything in glance and then tie it all together on an image basis, but smaller building blocks | |
| 16:37:28 | sean-k-mooney | TheJulia: well im not sure i agree that this should all live in glance or in the flavor | |
| 16:37:36 | sean-k-mooney | some of it should be user-data | |
| 16:38:00 | dansmith | it's not a smaller building block, it's a hammer to break the existing blocks | |
| 16:38:01 | dansmith | it doesn't build us toward a solution, it's an exit plan | |
| 16:38:43 | dansmith | instead of ever building new blocks, it's a forever solution of "get it to 40% complete and then do the rest custom out of band" | |
| 16:39:46 | sean-k-mooney | TheJulia: it sound like a way to pass info on a per instance basis to the ironic python agent would partly solve there issue. e.g. if the agent had a defien api of actions such as bios and raid config that could be pass on a per instance basis it would help with the image/flavor explosion issue | |
| 16:39:58 | TheJulia | woudl it help to have a larger discussion regarding this again, because where this always stalled out was the "how do we end up having something that nova will agree to" in the past in terms of data. and I think the last agreement of a possible solution involved templates stored in glance associated via traits or something which I think made it complex and cumbersome to interact with | |
| 16:40:19 | sean-k-mooney | the problem is that that for that to make sense it has to be somewhat abstract as we will not know the precise host ahead of time by design | |
| 16:40:55 | dansmith | we did have a long conversation about this in dublin, | |
| 16:41:03 | TheJulia | sean-k-mooney: we're actually have it, just turns out people want to do even more complex things their own way | |
| 16:41:10 | dansmith | which involved nova just knowing about a hardware template that gets passed to ironic to define these things | |
| 16:41:27 | dansmith | there was no tableflipping | |
| 16:41:54 | sean-k-mooney | dansmith: to do that we would want to do it in a way that did not invovle specifying the template in the flavor or image right | |