| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-23 | |||
| 12:25:54 | Spazmotic | Well devstack is a command line installer so many copy and paste the error from terminal into a github gist :) | |
| 12:27:02 | openstackgerrit | Elod Illes proposed openstack/nova master: Functional test: cold migrate to compute down https://review.openstack.org/496280 | |
| 12:29:52 | srf | Oke i'll try. | |
| 12:56:27 | jaypipes | bauwser: is sahid going to PTG? | |
| 13:00:53 | bauwser | leakypipes: no, mostly all our team but him :) | |
| 13:03:18 | leakypipes | bauwser: shame :( there's a bunch of specs I wanted to discuss with him. | |
| 13:05:27 | leakypipes | dansmith: still around? | |
| 13:05:39 | leakypipes | dansmith: question for you on that IO thread vs vCPU thread thing... | |
| 13:06:15 | sean-k-mooney | leakypipes: e.g. qemu emulator treads? | |
| 13:06:28 | sean-k-mooney | they are the ones that od io right? | |
| 13:07:05 | leakypipes | sean-k-mooney: are you making fun of me? :) | |
| 13:07:54 | sean-k-mooney | haha no it geunely works differently depend on kernel vs dpdk vhost or kvm vs qeum with tci backend so it never keep this strait in my head | |
| 13:08:12 | leakypipes | sean-k-mooney: no, AFAIK, the emulator threads are the QEMU control process threads -- they send communication events to the guest I think? | |
| 13:08:54 | leakypipes | sean-k-mooney: the I/O threads are for doing device read/write operations on behalf of the guest. And the vCPU threads are for the guest's userspace code to run. | |
| 13:09:45 | leakypipes | sean-k-mooney: but I had a question about whether libvirt allows each of those three types of threads to be pinned to a specific pCPU or whether only emulator and vCPU threads were possible to pin... | |
| 13:10:15 | johnthetubaguy | you know I assumed it pinned them all together, but that is a good question | |
| 13:10:33 | leakypipes | sean-k-mooney: and further to that, can you pin the emulator thread(s) to the same pCPU as the I/O threads... | |
| 13:12:34 | sahid | leakypipes: you can pin i/o threads with emulator threads on a same pCPUs but it can have bad effects, and increase I/O lantency | |
| 13:12:57 | leakypipes | sahid: ok, cool. thanks :) | |
| 13:13:15 | leakypipes | sahid: the emulator thread doesn't get a whole lot of control process "traffic" though, right? | |
| 13:13:26 | leakypipes | sahid: pretty minimal compared to IO or vCPU threads, yeah? | |
| 13:14:05 | leakypipes | sahid: I mean, I understand for RT environments wanting to have guaranteed execution and all that, I'm just curious about the internals of it all, nothing more. | |
| 13:15:35 | sahid | honnestly i don't know exaclty, it's just that we know the emulator threads can lock the full CPU so we want avoid to have vCPUs running sensitive apps | |
| 13:16:12 | sahid | i would say we want the same for i/o threads so it's probably better to avoid to pin them together | |
| 13:16:43 | leakypipes | got it. | |
| 13:16:46 | leakypipes | thanks :) | |
| 13:21:04 | srf | i'll try to install devstack in my computer but always error in nova http://paste.openstack.org/show/682946/ anybody can help me what wrong this | |
| 13:24:53 | hshiina | leakypipes: hi, i have a question about your note at https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L855 | |
| 13:25:19 | hshiina | InventoryInUse exception is periodically logged with stacktrace in ironic job: http://logs.openstack.org/19/546919/2/check/ironic-tempest-dsvm-ipa-partition-pxe_ipmitool-tinyipa-python3/2737ab0/logs/screen-n-cpu.txt.gz?level=ERROR | |
| 13:25:45 | hshiina | is there anything to fix here? | |
| 13:33:32 | leakypipes | hshiina: one sec, reading :) | |
| 13:35:38 | leakypipes | hshiina: no, that looks to be a different scenario than the one described in that comment. lemme dig some more into the log there. | |
| 13:37:30 | hshiina | leakypipes: sure, thanks. | |
| 13:37:45 | leakypipes | hshiina: if I had to guess, what is happening in your case is the following: | |
| 13:37:57 | leakypipes | 1) nova boots an Ironic flavor | |
| 13:38:37 | leakypipes | 2) nova-scheduler finds an available Ironic compute node that has CUSTOM_BAREMETAL inventory of 1, used of 0. | |
| 13:38:52 | leakypipes | 3) nova-compute calls the Ironic virt driver to launch the instance | |
| 13:39:09 | leakypipes | 4) Ironic virt driver launches the instance. All good. | |
| 13:39:24 | leakypipes | 5) At some point, the Ironic node goes into an "unavailable" status | |
| 13:39:36 | leakypipes | 6) The resource tracker in nova-compute runs update_available_resource() | |
| 13:39:56 | leakypipes | 7) the Ironic virt driver returns an inventory total of 0 CUSTOM_BAREMETAL for the unavailable node | |
| 13:40:12 | leakypipes | 8) The nova-compute resource tracker dutifully tries to remove the inventory for that node | |
| 13:40:31 | leakypipes | 9) the Placement API returns an InventoryInUse exception because there is an instance running on that Ironic node | |
| 13:41:09 | leakypipes | hshiina: so what I would check in your job logs is whether that Ironic node that the instance ended up on was set to unavailable at some point between 11:17:05 and 11:18:13 | |
| 13:41:57 | leakypipes | hshiina: from what I can tell, the resource tracker and placement API are doing "the right thing" and preventing inventory from being deleted when there is an active instance on that Ironic node. | |
| 13:43:30 | hshiina | leakypipes: thank you. i will dig it later. | |
| 13:43:50 | hshiina | leakypipes: i am interested in. but, i have to leave office shortly. | |
| 13:43:59 | leakypipes | np | |
| 13:50:58 | srf | i'll try to install devstack in my computer but always error in nova http://paste.openstack.org/show/682946/ anybody can help me, ? | |
| 13:54:03 | openstack | Launchpad bug 1726260 in devstack "devstack fails while running ./stack.sh in master/queens in ubuntu 16.04 while starting n-cpu" [Undecided,New] | |
| 13:54:03 | Spazmotic | srf: Quick google revealed this.. did you try to search around a bit? https://bugs.launchpad.net/devstack/+bug/1726260 | |
| 13:54:26 | Spazmotic | Give that a try :) | |
| 13:58:14 | srf | Spazmotic : I've tried to adding ENABLED_SERVICES=placement-api but still error | |
| 14:05:33 | Spazmotic | Not too sure then, but i'm sure someone may have seen it before.. also again you may wish to ask in #openstack as that's more designed for deployment support | |
| 14:24:20 | Guest70 | hello | |
| 14:44:17 | openstackgerrit | Eric Berglund proposed openstack/nova master: WIP: PowerVM Driver: Network interface attach/detach https://review.openstack.org/546813 | |
| 14:45:08 | openstackgerrit | Theodoros Tsioutsias proposed openstack/nova master: WIP: Add the reaper entry point https://review.openstack.org/547450 | |
| 14:53:14 | bauwser | if folks have specs, I'm hungry to eat them | |
| 14:53:44 | bauwser | like, you're going to the PTG and would like to discuss about your spec, I'm your man | |
| 14:54:16 | figleaf | bauwser: after you eat them, you know what they turn into :) | |
| 14:54:44 | bauwser | if they are made of cheese, I'm +2 | |
| 14:55:16 | bauwser | even if https://www.thelocal.fr/20180223/french-cheese-wars-are-the-days-of-the-real-normandy-camembert-numbered | |
| 14:57:20 | figleaf | guess it will be like taking a microbrew beer and then mass-producing it | |
| 15:04:39 | dansmith | leakypipes: I'm really not the right person to answer detailed questions about that | |
| 15:15:33 | bauwser | dansmith: I don't know your current TZ, but it's Friday here :) | |
| 15:15:54 | dansmith | bauwser: I don't know my current TZ either, but I'm pretty sure it's not friday in my body | |
| 15:16:04 | bauwser | still on the air ? | |
| 15:16:16 | dansmith | no, you caught me, I know my current TZ | |
| 15:16:19 | dansmith | but my body doesn't | |
| 15:16:22 | bauwser | heh | |
| 15:19:11 | mriedem | why do we have https://github.com/openstack/nova/blob/ef4000a0d326deb004843ee51d18030224c5630f/nova/db/sqlalchemy/api_models.py#L409 ? | |
| 15:19:24 | mriedem | seems silly to have an entire table to hold one string | |
| 15:19:40 | mriedem | maybe it goes back to the idea that server groups would have >1 policy | |
| 15:19:42 | mriedem | but the API doesn't allow that | |
| 15:20:26 | mriedem | https://github.com/openstack/nova/blob/ef4000a0d326deb004843ee51d18030224c5630f/nova/api/openstack/compute/schemas/server_groups.py#L29-L39 | |
| 15:21:42 | dansmith | mriedem: yeah it's 1:N I think | |
| 15:21:47 | dansmith | not that we can do that, I don't think | |
| 15:22:09 | mriedem | the api doesn't allow it | |
| 15:22:14 | mriedem | but that's the legacy reason i guess, | |
| 15:22:23 | mriedem | the schema makes you pass a single-item list for the policy | |
| 15:22:44 | dansmith | but even still, none of the stuff we do to honor the policy considers multiple options right? | |
| 15:22:47 | mriedem | and it's called "policies" but doesn't allow >1 | |
| 15:22:54 | mriedem | correct | |
| 15:22:56 | mriedem | it's just weird | |
| 15:23:06 | leakypipes | cfriesen: am I gonna have to get out the beating stick for you? :P | |
| 15:23:26 | mriedem | dansmith: context is https://review.openstack.org/#/c/546925 which is tied to an old blueprint from sgordon | |
| 15:23:45 | leakypipes | cfriesen: well, I suppose we could rent a pair of those fake Sumo wrestler costumes and go at it. | |
| 15:25:02 | dansmith | mriedem: ah yes, affrinity | |
| 15:25:18 | mriedem | i'm currently in the process of ripping up the spec | |
| 15:26:00 | dansmith | mriedem: ugh, that max_number_per_host thing is just silly.. that's never going to do something useful, IMHO | |
| 15:27:58 | mriedem | i can see the use, | |
| 15:28:03 | mriedem | it makes the soft policy less soft | |
| 15:28:08 | mriedem | jello-like if you will | |
| 15:28:19 | dansmith | what's the point in that? | |
| 15:28:41 | mriedem | there is some nfv use case in the spec | |
| 15:29:04 | dansmith | I think from a modeling point of view, it'd be hard to reason about that unless you knew "I'm going to start ten of the exact same thing for throughput and I want those spread across three nodes" | |
| 15:29:20 | mriedem | anti-affinity to a point, so they don't need all VMs in the group on separate hosts, but they can only tolerate up to a limit of VMs on the same host | |
| 15:30:14 | mriedem | otherwise today if you have 10 VMs in a soft-anti-affinity group, and only 2 available hosts, you could get like 1 on 1 host and 9 on the other | |
| 15:30:19 | dansmith | right, so a more useful thing would be to say "instances in this group should spread across N hosts before considering doubling up" | |
| 15:30:31 | dansmith | that's an easier to reason-about model from the outside I think | |