| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-08-23 | |||
| 17:25:43 | openstackgerrit | Merged openstack/nova stable/rocky: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677176 | |
| 17:25:50 | openstackgerrit | Merged openstack/nova stable/queens: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677177 | |
| 17:25:57 | openstackgerrit | Merged openstack/nova stable/pike: Remove experimental job on openSUSE 42.3 https://review.opendev.org/677178 | |
| 17:28:37 | efried | mriedem, dansmith: Posted details on the SIGHUP patch https://review.opendev.org/#/c/641907/ | |
| 17:29:01 | efried | We are indeed breaking the privsep pipe (/me says as if I know what I'm talking about) | |
| 17:29:15 | efried | unclear where that would need to be fixed. | |
| 17:29:42 | mriedem | stephenfin: yeah irrelevant-files | |
| 17:36:47 | sean-k-mooney | stephenfin: add a followup [DNM] patch that add whitespace to a file that would trigger tempest or temporally remove one fo the fiels from irrelevent-files. i generall prefer to have a DNM testing patch as i can just run the job i care about. | |
| 17:37:39 | mriedem | efried: mnaser: i think we likely should undeprecate the direct file download stuff since clearly there are users and those users have made attempts to upstream an alternative solution which hasn't been approved, but not wholesale rejected either from what i understand, so it's probably not good to just rip it out from them when we're not providing an alternative | |
| 17:38:16 | efried | wfm | |
| 17:38:32 | sean-k-mooney | stephenfin: that chante will alter they way devstack install nova as nova installs all packages in requirements.txt and test-requiremnts.txt system wide by default so it should pass in the gate | |
| 17:38:59 | sean-k-mooney | stephenfin: however if you enable installing in virtual envs it may not work anymore | |
| 17:39:53 | sean-k-mooney | stephenfin: without a change to devstack nova would be relying on other pojects to install pymsql or psycopg2 for postgress | |
| 17:43:14 | efried | mriedem: any advice on next steps for copying mysql logs? https://review.opendev.org/#/c/678051/ | |
| 17:45:54 | efried | mriedem: | |
| 17:45:54 | efried | Warning: Permanently added '166.78.116.9' (ECDSA) to the list of known hosts. | |
| 17:45:54 | efried | sudo: no tty present and no askpass program specified | |
| 17:45:54 | efried | rsync: connection unexpectedly closed (0 bytes received so far) [Receiver] | |
| 17:45:54 | efried | rsync error: error in rsync protocol data stream (code 12) at io.c(226) [Receiver=3.1.1] | |
| 17:47:12 | sean-k-mooney | i dont think you want to copy them to the executer liek that i think you want to drop them into the working directory on the vm and have the normal log copy task copy them at the end | |
| 17:48:14 | sean-k-mooney | i may be wrong about that but i think that is what i have seen done before | |
| 17:49:53 | artom | sean-k-mooney, do you know if we would break anything if we started doing claims for all live migrations? I'm thinking of PCI stuff... | |
| 17:50:13 | artom | Presumably claims for PCI aren't broken in and of themselves, since they're used to boot instances and for cold migrations | |
| 17:50:31 | sean-k-mooney | we do claims for pci deivces for migration already | |
| 17:50:40 | artom | *live*? | |
| 17:50:43 | sean-k-mooney | yes | |
| 17:51:07 | sean-k-mooney | well for neutron sriov ports | |
| 17:51:17 | sean-k-mooney | we dont support live migration with other pci devices | |
| 17:51:30 | sean-k-mooney | but for sriov migration we added cliams | |
| 17:51:45 | sean-k-mooney | *sriov live migration | |
| 17:51:51 | artom | Are we talking about the same claims? | |
| 17:51:59 | sean-k-mooney | in the resouce tracker | |
| 17:52:03 | artom | Yeah | |
| 17:52:10 | sean-k-mooney | we do it as part of pre live migration at dest | |
| 17:52:11 | artom | Could you point me to the review/code? | |
| 17:52:22 | sean-k-mooney | lookign for it now | |
| 17:53:30 | sean-k-mooney | https://review.opendev.org/#/c/616120/ | |
| 17:53:41 | sean-k-mooney | and i think we use them in https://review.opendev.org/#/c/620115/28 | |
| 17:54:32 | sean-k-mooney | the pci manager works differently then the numa toplogy in the RT | |
| 17:54:59 | sean-k-mooney | but ehere is the claim in the compute manager | |
| 17:55:01 | sean-k-mooney | https://review.opendev.org/#/c/620115/28/nova/compute/manager.py | |
| 17:55:45 | openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | |
| 17:55:45 | openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | |
| 17:56:43 | artom | sean-k-mooney, it's not actually creating a MoveClaim object though, is it? | |
| 17:57:33 | artom | sean-k-mooney, I'm thinking of a scenario where we live-migration a NUMA instance with neutron SRIOV ports | |
| 17:57:50 | sean-k-mooney | no its not as far as i rememeber | |
| 17:57:53 | artom | Will the MoveClaim for the NUMA part conflict with the PCI claim for the SRIOV part | |
| 17:58:23 | sean-k-mooney | no we claim them in the rt and store the pci addresses in the live migration data object | |
| 17:58:31 | sean-k-mooney | i dont think we are using move claims at all | |
| 17:59:46 | sean-k-mooney | we might use move claims for cold migration i have not check that in a long time | |
| 17:59:52 | mriedem | efried: i think it's failing because the dest doesn't exist when your task runs | |
| 18:00:03 | mriedem | that's done later here https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/5ba0f501-9e58-4f71-859a-ef084a19e0be/#line-11 | |
| 18:00:33 | efried | mriedem: infra said it was because sudo is disabled for tox by default. Latest respin (above) reparents to openstack-tox-with-sudo. | |
| 18:00:33 | artom | sean-k-mooney, if you have the hardware to test those 2 together it would put many minds at east, I think | |
| 18:00:58 | sean-k-mooney | i plan to do that yes | |
| 18:01:20 | sean-k-mooney | ill do it monday or tuesday if you like | |
| 18:01:34 | efried | mriedem: but maybe that too. Though I would have thought they would set up log dirs before invoking posts. | |
| 18:02:50 | sean-k-mooney | i have 1 of my sriov systems setup with 18.04 so i need to install 18.04 on the second one an then install centos on a third host to test something else | |
| 18:03:19 | sean-k-mooney | ill kick of those now and i should be able to test it monday if you bug me or tuesday if you dont | |
| 18:05:39 | artom | sean-k-mooney, appreciated :) | |
| 18:06:30 | mriedem | efried: maybe it is https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/c8a99499-4049-4878-b0a9-13a9292437f4/#line-1 | |
| 18:06:48 | mriedem | efried: so maybe your dest needs to be '{{ zuul_log_path }}' | |
| 18:06:53 | mriedem | er + /mysql/ | |
| 18:07:24 | mriedem | or {{ zuul_output_dir }}/logs/mysql | |
| 18:07:28 | mriedem | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/3/check/nova-tox-collect-py27/f64b3ba/ara-report/file/f31abc72-92ae-4e7d-b37c-8c124f7f32bd/#line-1 | |
| 18:09:55 | efried | mriedem: okay, can try something along those lines if this one borks again. | |
| 18:10:25 | mriedem | ok | |
| 18:10:31 | mriedem | you know about the ara report right? | |
| 18:10:39 | mriedem | to see what runs before and after you with what values | |
| 18:13:52 | sean-k-mooney | artom: once i have it up an running i can give you remote acess to poke around if you want. i have a static ip so i can easily port forwad you to i the hosts. ubuntu is install now so i just need to add a couple of kernel arges to make sriov work then stack with your code | |
| 18:14:07 | artom | sean-k-mooney, ack, that works | |
| 18:15:20 | artom | sean-k-mooney, I'm thinking of this bit in the resource tracker: https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L283-L295 | |
| 18:15:35 | artom | For NUMA LM, we call that during check_live_migration_at_destination | |
| 18:16:47 | efried | mriedem: no, I knew it was there, but never really understood what it meant. | |
| 18:18:55 | mriedem | btw, speaking of that nova.image.download module https://logs.opendev.org/84/676684/1/check/neutron-grenade-multinode/b0a76ac/logs/screen-n-api.txt#_Aug_15_13_25_04_744293 | |
| 18:18:59 | mriedem | i'm not sure why that's happening | |
| 18:19:05 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/compute/resource_tracker.py#L294-L295 might be a proablem | |
| 18:19:31 | sean-k-mooney | since we have already claimed them on the dest it could claim them twice | |
| 18:19:50 | mriedem | oh gdi of course https://review.opendev.org/#/c/676684/ | |
| 18:19:51 | mriedem | hehh | |
| 18:19:56 | sean-k-mooney | i think currenlty we only use that for cold migration right | |
| 18:21:30 | artom | sean-k-mooney, *currently* | |
| 18:21:45 | artom | But NUMA LM is based on doing a MoveClaim | |
| 18:22:09 | sean-k-mooney | right so we would have to modify this or it wont work with sriov live migration | |
| 18:22:18 | sean-k-mooney | or rather might not | |
| 18:23:18 | sean-k-mooney | will you be passing in somthing to indicate its a live migration | |
| 18:23:20 | artom | Or just ignore pci_requests entirely if it's a live migration? | |
| 18:23:31 | sean-k-mooney | that is what i was going to suggest | |
| 18:23:33 | artom | Yeah, the migration will hava a type set | |
| 18:23:34 | openstackgerrit | Andreas Jaeger proposed openstack/nova master: PDF documentation build https://review.opendev.org/676730 | |
| 18:23:54 | sean-k-mooney | its nice that it is claiming alias based pci devices | |
| 18:24:02 | sean-k-mooney | we dont currently support that in the sriov migration | |
| 18:24:13 | sean-k-mooney | but that also means we woudl not update the xml | |
| 18:24:43 | artom | Yeah, we can't claim resource we have no intention of using | |
| 18:24:44 | sean-k-mooney | so what we need to to is 1 check that there is not alias based pci device and two not call the cpi claim bit if its a live migration | |
| 18:25:06 | artom | Why is 2 alone not enough? | |
| 18:25:30 | sean-k-mooney | it is but the other is a safty mesure | |
| 18:25:48 | sean-k-mooney | e.g. we shoudl not attemt to migrate if you have alias based pci passthough | |
| 18:25:54 | sean-k-mooney | its not supported at all | |
| 18:26:08 | sean-k-mooney | we might already check that somewhere | |